2024-04-24T23:41:58.611 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T23:41:58.611 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/recovery-fs} --verbose --archive /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219 --name teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219/orig.config.yaml 2024-04-24T23:41:58.657 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T23:41:58.750 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219 branch: squid description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/recovery-fs} email: ceph-qa@ceph.com first_in_suite: false job_id: '7672219' kernel: kdb: 1 sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 8528 sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 sleep_before_teardown: 0 subset: 122/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYsIB19cMcfrT6kPr3H+ZQvTnJ3B601Az23vEK1M8i2MEhX932Bvugfhcgl2rvWXqSCW79GONnaiWvJvY6YON0= smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMnVq9hvlZNZZI3TLOcEDPIl6FQneeSABfDPoVkLzqN/MwqPBV2bVTfvrQtQTERyEMFWiEALYEF4zj68ooehLOo= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_recovery_fs teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T23:41:58.750 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa; will attempt to use it 2024-04-24T23:41:58.751 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa/tasks 2024-04-24T23:41:58.751 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T23:41:58.752 INFO:teuthology.task.internal:Checking packages... 2024-04-24T23:41:58.782 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd2c45d977142e024a59b68a8f1dce8f0b9840551' 2024-04-24T23:41:58.783 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T23:41:58.783 INFO:teuthology.packaging:ref: None 2024-04-24T23:41:58.783 INFO:teuthology.packaging:tag: None 2024-04-24T23:41:58.783 INFO:teuthology.packaging:branch: squid 2024-04-24T23:41:58.783 INFO:teuthology.packaging:sha1: d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:41:58.783 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-24T23:41:59.014 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2445.gd2c45d97 2024-04-24T23:41:59.016 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T23:41:59.022 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T23:41:59.022 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T23:41:59.034 INFO:teuthology.task.internal:Saving configuration 2024-04-24T23:41:59.053 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T23:41:59.059 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T23:41:59.085 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 23:36:59.112312', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYsIB19cMcfrT6kPr3H+ZQvTnJ3B601Az23vEK1M8i2MEhX932Bvugfhcgl2rvWXqSCW79GONnaiWvJvY6YON0='} 2024-04-24T23:41:59.103 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 23:36:59.114313', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMnVq9hvlZNZZI3TLOcEDPIl6FQneeSABfDPoVkLzqN/MwqPBV2bVTfvrQtQTERyEMFWiEALYEF4zj68ooehLOo='} 2024-04-24T23:41:59.103 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T23:41:59.109 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-04-24T23:41:59.109 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-04-24T23:41:59.109 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T23:41:59.185 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6f7346e0d0>, signals=[15]) 2024-04-24T23:41:59.185 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T23:41:59.191 INFO:teuthology.task.internal:Opening connections... 2024-04-24T23:41:59.191 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:41:59.194 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:41:59.288 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:41:59.289 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:41:59.373 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T23:41:59.380 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-04-24T23:41:59.399 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-04-24T23:41:59.400 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-04-24T23:41:59.460 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-04-24T23:41:59.460 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-04-24T23:41:59.460 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-04-24T23:41:59.460 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-04-24T23:41:59.460 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-04-24T23:41:59.460 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-04-24T23:41:59.460 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T23:41:59.460 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-04-24T23:41:59.460 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-04-24T23:41:59.460 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T23:41:59.460 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-04-24T23:41:59.461 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T23:41:59.461 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T23:41:59.461 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T23:41:59.461 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-04-24T23:41:59.487 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2024-04-24T23:41:59.507 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2024-04-24T23:41:59.507 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2024-04-24T23:41:59.567 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Stream" 2024-04-24T23:41:59.567 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="9" 2024-04-24T23:41:59.567 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2024-04-24T23:41:59.567 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2024-04-24T23:41:59.568 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="9" 2024-04-24T23:41:59.568 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el9" 2024-04-24T23:41:59.568 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T23:41:59.568 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2024-04-24T23:41:59.568 INFO:teuthology.orchestra.run.smithi156.stdout:LOGO="fedora-logo-icon" 2024-04-24T23:41:59.568 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T23:41:59.568 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://centos.org/" 2024-04-24T23:41:59.568 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T23:41:59.568 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T23:41:59.568 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T23:41:59.568 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2024-04-24T23:41:59.588 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T23:41:59.645 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T23:41:59.718 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T23:41:59.718 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-04-24T23:41:59.721 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2024-04-24T23:41:59.737 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T23:41:59.743 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T23:41:59.744 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-04-24T23:41:59.779 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2024-04-24T23:41:59.877 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T23:41:59.927 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T23:41:59.944 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-24T23:41:59.944 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-24T23:41:59.944 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T23:41:59.945 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T23:41:59.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:41:59.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:41:59.962 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2024-04-24T23:41:59.963 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-04-24T23:42:00.019 INFO:teuthology.orchestra.run.smithi156.stdout:5.14.0-437.el9.x86_64 2024-04-24T23:42:00.019 INFO:teuthology.task.kernel:Running kernel on smithi156: 5.14.0-437.el9.x86_64 2024-04-24T23:42:00.019 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum install -y kernel 2024-04-24T23:42:00.020 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-437.el9.x86_64 2024-04-24T23:42:00.020 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.14.0-437.el9.x86_64 2024-04-24T23:42:00.021 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2024-04-24T23:42:02.836 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 7.0 MB/s | 8.0 MB 00:01 2024-04-24T23:42:04.696 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-04-24T23:42:07.265 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 1.4 MB/s | 8.0 MB 00:05 2024-04-24T23:42:08.908 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 33 MB/s | 19 MB 00:00 2024-04-24T23:42:10.346 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 6.2 MB 00:01 2024-04-24T23:42:12.647 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-04-24T23:42:13.709 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 5.9 MB/s | 6.2 MB 00:01 2024-04-24T23:42:14.173 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-04-24T23:42:15.334 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-04-24T23:42:16.386 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-04-24T23:42:19.415 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-24T23:42:21.363 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-24T23:42:21.363 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-24T23:42:21.468 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-24T23:42:21.469 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-04-24T23:42:21.469 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-24T23:42:21.485 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-24T23:42:21.570 DEBUG:teuthology.orchestra.run.smithi016:> echo no | sudo yum reinstall kernel || true 2024-04-24T23:42:22.131 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 11:42:19 PM UTC. 2024-04-24T23:42:22.263 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-24T23:42:22.278 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-24T23:42:22.278 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-24T23:42:22.278 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-24T23:42:22.278 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalling: 2024-04-24T23:42:22.278 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T23:42:22.279 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:42:22.279 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-24T23:42:22.279 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-24T23:42:22.279 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:42:22.279 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 255 k 2024-04-24T23:42:22.279 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 0 2024-04-24T23:42:22.279 INFO:teuthology.orchestra.run.smithi016.stderr:Operation aborted. 2024-04-24T23:42:22.331 INFO:teuthology.orchestra.run.smithi016.stdout:Is this ok [y/N]: 2024-04-24T23:42:22.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum reinstall -y kernel || true 2024-04-24T23:42:22.845 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 11:42:19 PM UTC. 2024-04-24T23:42:22.966 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-24T23:42:22.967 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-24T23:42:22.967 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-04-24T23:42:22.967 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-24T23:42:22.967 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalling: 2024-04-24T23:42:22.967 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T23:42:22.967 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:42:22.967 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-24T23:42:22.968 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-04-24T23:42:22.968 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:42:22.968 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 255 k 2024-04-24T23:42:22.968 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 0 2024-04-24T23:42:22.968 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-24T23:42:23.359 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-24T23:42:23.378 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-24T23:42:23.452 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-04-24T23:42:23.453 INFO:teuthology.orchestra.run.smithi156.stdout:Nothing to do. 2024-04-24T23:42:23.453 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-04-24T23:42:23.576 DEBUG:teuthology.orchestra.run.smithi156:> echo no | sudo yum reinstall kernel || true 2024-04-24T23:42:23.584 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-437.el9.x86_64.rpm 464 kB/s | 255 kB 00:00 2024-04-24T23:42:23.584 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-24T23:42:23.586 INFO:teuthology.orchestra.run.smithi016.stdout:Total 413 kB/s | 255 kB 00:00 2024-04-24T23:42:23.600 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-24T23:42:23.800 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-24T23:42:23.800 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-24T23:42:23.804 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-24T23:42:23.818 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-24T23:42:23.935 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-24T23:42:24.002 INFO:teuthology.orchestra.run.smithi016.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T23:42:24.002 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T23:42:24.121 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 11:42:21 PM UTC. 2024-04-24T23:42:24.241 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-04-24T23:42:24.258 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-24T23:42:24.258 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2024-04-24T23:42:24.258 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-24T23:42:24.258 INFO:teuthology.orchestra.run.smithi156.stdout:Reinstalling: 2024-04-24T23:42:24.258 INFO:teuthology.orchestra.run.smithi156.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T23:42:24.259 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:42:24.259 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-04-24T23:42:24.259 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-24T23:42:24.259 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:42:24.259 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 255 k 2024-04-24T23:42:24.260 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 0 2024-04-24T23:42:24.260 INFO:teuthology.orchestra.run.smithi156.stderr:Operation aborted. 2024-04-24T23:42:24.321 INFO:teuthology.orchestra.run.smithi156.stdout:Is this ok [y/N]: 2024-04-24T23:42:24.322 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum reinstall -y kernel || true 2024-04-24T23:42:24.739 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T23:42:24.740 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T23:42:24.820 INFO:teuthology.orchestra.run.smithi156.stdout:Last metadata expiration check: 0:00:03 ago on Wed 24 Apr 2024 11:42:21 PM UTC. 2024-04-24T23:42:24.936 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-04-24T23:42:24.938 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-24T23:42:24.938 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2024-04-24T23:42:24.938 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-24T23:42:24.938 INFO:teuthology.orchestra.run.smithi156.stdout:Reinstalling: 2024-04-24T23:42:24.938 INFO:teuthology.orchestra.run.smithi156.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-24T23:42:24.938 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:42:24.938 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-04-24T23:42:24.939 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-04-24T23:42:24.939 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:42:24.939 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 255 k 2024-04-24T23:42:24.939 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 0 2024-04-24T23:42:24.939 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-04-24T23:42:25.221 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.7 MB/s | 255 kB 00:00 2024-04-24T23:42:25.222 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-04-24T23:42:25.222 INFO:teuthology.orchestra.run.smithi156.stdout:Total 898 kB/s | 255 kB 00:00 2024-04-24T23:42:25.243 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-04-24T23:42:25.431 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-04-24T23:42:25.432 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-04-24T23:42:25.437 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-04-24T23:42:25.437 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-04-24T23:42:25.443 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T23:42:25.443 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:42:25.444 INFO:teuthology.orchestra.run.smithi016.stdout:Reinstalled: 2024-04-24T23:42:25.444 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-24T23:42:25.444 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:42:25.444 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-24T23:42:25.576 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-04-24T23:42:25.598 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T23:42:25.647 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-24T23:42:25.647 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-24T23:42:25.647 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-24T23:42:25.648 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-24T23:42:25.648 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T23:42:25.655 INFO:teuthology.orchestra.run.smithi156.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T23:42:25.656 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T23:42:25.688 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2024-04-24T23:42:25.707 DEBUG:teuthology.parallel:result is None 2024-04-24T23:42:26.336 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T23:42:26.337 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-24T23:42:27.054 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-24T23:42:27.054 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:42:27.055 INFO:teuthology.orchestra.run.smithi156.stdout:Reinstalled: 2024-04-24T23:42:27.055 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-24T23:42:27.055 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:42:27.055 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-04-24T23:42:27.236 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2024-04-24T23:42:27.283 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-24T23:42:27.284 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-24T23:42:27.284 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-24T23:42:27.284 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-24T23:42:27.284 DEBUG:teuthology.orchestra.run.smithi156:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-24T23:42:27.317 INFO:teuthology.orchestra.run.smithi156.stdout:ttyS1 2024-04-24T23:42:27.341 DEBUG:teuthology.parallel:result is None 2024-04-24T23:42:27.341 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T23:42:27.349 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T23:42:27.350 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T23:42:27.353 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T23:42:27.396 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T23:42:27.403 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T23:42:27.409 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T23:42:27.409 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T23:42:27.412 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T23:42:27.464 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T23:42:27.470 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T23:42:27.470 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T23:42:27.490 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T23:42:27.531 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:42:27.532 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:42:27.540 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:42:27.543 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T23:42:27.545 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T23:42:27.552 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T23:42:27.552 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T23:42:27.555 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T23:42:27.605 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T23:42:27.615 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T23:42:27.615 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T23:42:27.623 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T23:42:27.661 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:42:27.716 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:42:27.773 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:42:27.851 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:42:27.896 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:42:27.897 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T23:42:27.962 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:42:28.001 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T23:42:28.049 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:42:28.125 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T23:42:28.170 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:42:28.170 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T23:42:28.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-04-24T23:42:28.236 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2024-04-24T23:42:28.268 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:42:28.305 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:42:28.778 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T23:42:28.788 INFO:teuthology.task.internal:Starting timer... 2024-04-24T23:42:28.788 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T23:42:28.798 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T23:42:28.806 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-04-24T23:42:28.807 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-04-24T23:42:28.876 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-04-24T23:42:28.878 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2024-04-24T23:42:28.916 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2024-04-24T23:42:28.919 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T23:42:28.919 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-04-24T23:42:28.950 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-04-24T23:42:28.950 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2024-04-24T23:42:28.982 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2024-04-24T23:42:28.982 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi156.front.sepia.ceph.com': 'permissive'} 2024-04-24T23:42:28.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:42:29.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:42:29.015 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:42:29.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:42:29.047 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T23:42:29.047 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-04-24T23:42:29.080 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2024-04-24T23:42:29.111 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T23:42:29.121 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T23:42:29.137 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T23:42:29.138 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi156.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T23:46:00.505 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi156.front.sepia.ceph.com')] 2024-04-24T23:46:00.507 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-24T23:46:00.508 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:46:00.587 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-04-24T23:46:00.670 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-04-24T23:46:00.670 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-04-24T23:46:00.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T23:46:00.743 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-04-24T23:46:00.822 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-04-24T23:46:00.822 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T23:46:00.833 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T23:46:00.833 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T23:46:00.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:46:00.836 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T23:46:00.836 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:46:00.862 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T23:46:00.876 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T23:46:00.890 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T23:46:00.899 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-04-24T23:46:00.903 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T23:46:00.910 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-04-24T23:46:00.924 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T23:46:00.926 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2024-04-24T23:46:00.936 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2024-04-24T23:46:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T23:46:00.950 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T23:46:00.963 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T23:46:00.992 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-04-24T23:46:00.996 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:46:00.996 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-04-24T23:46:00.996 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:46:00.996 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:46:00.996 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:46:00.996 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:46:01.023 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-04-24T23:46:01.026 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:46:01.026 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-04-24T23:46:01.026 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:46:01.026 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:46:01.027 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:46:01.027 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:46:01.028 INFO:teuthology.run_tasks:Running task install... 2024-04-24T23:46:01.039 DEBUG:teuthology.task.install:project ceph 2024-04-24T23:46:01.039 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}} 2024-04-24T23:46:01.039 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'} 2024-04-24T23:46:01.039 INFO:teuthology.task.install:Using flavor: default 2024-04-24T23:46:01.045 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T23:46:01.045 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-24T23:46:01.046 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-24T23:46:01.046 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:46:01.048 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'tag': None, 'wait_for_package': False} 2024-04-24T23:46:01.049 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:46:01.272 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-24T23:46:01.272 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-24T23:46:01.277 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/ 2024-04-24T23:46:01.277 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2445.gd2c45d97 2024-04-24T23:46:01.553 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T23:46:01.554 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:46:01.554 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T23:46:01.579 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T23:46:01.579 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T23:46:01.605 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/d2c45d977142e024a59b68a8f1dce8f0b9840551/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T23:46:01.605 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:46:01.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T23:46:01.631 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T23:46:01.632 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d2c45d977142e024a59b68a8f1dce8f0b9840551/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T23:46:01.643 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T23:46:01.698 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T23:46:01.717 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T23:46:01.773 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T23:46:01.780 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2024-04-24T23:46:01.781 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2024-04-24T23:46:01.837 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-04-24T23:46:01.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-04-24T23:46:02.006 INFO:teuthology.orchestra.run.smithi156.stdout:48 files removed 2024-04-24T23:46:02.034 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T23:46:02.074 INFO:teuthology.orchestra.run.smithi016.stdout:48 files removed 2024-04-24T23:46:02.101 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T23:46:02.723 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 194 kB/s | 83 kB 00:00 2024-04-24T23:46:02.849 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 175 kB/s | 83 kB 00:00 2024-04-24T23:46:03.157 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 29 kB/s | 11 kB 00:00 2024-04-24T23:46:03.300 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 28 kB/s | 11 kB 00:00 2024-04-24T23:46:03.566 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-04-24T23:46:03.725 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-04-24T23:46:04.166 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-24T23:46:04.575 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 9.9 MB/s | 8.0 MB 00:00 2024-04-24T23:46:06.526 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-04-24T23:46:10.694 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.2 MB 00:00 2024-04-24T23:46:12.087 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 16 kB 00:00 2024-04-24T23:46:14.380 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 9.6 MB/s | 21 MB 00:02 2024-04-24T23:46:19.738 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-04-24T23:46:21.941 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T23:46:21.941 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T23:46:21.952 INFO:teuthology.orchestra.run.smithi016.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T23:46:21.953 INFO:teuthology.orchestra.run.smithi016.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T23:46:21.954 INFO:teuthology.orchestra.run.smithi016.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T23:46:21.955 INFO:teuthology.orchestra.run.smithi016.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T23:46:21.956 INFO:teuthology.orchestra.run.smithi016.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T23:46:21.959 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T23:46:21.959 INFO:teuthology.orchestra.run.smithi016.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T23:46:22.025 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-04-24T23:46:22.035 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-04-24T23:46:22.035 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-04-24T23:46:22.035 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-04-24T23:46:22.035 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-04-24T23:46:22.035 INFO:teuthology.orchestra.run.smithi016.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T23:46:22.035 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-24T23:46:22.035 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-24T23:46:22.035 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-24T23:46:22.035 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-24T23:46:22.035 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-24T23:46:22.035 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-24T23:46:22.036 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-24T23:46:22.036 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-24T23:46:22.036 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-24T23:46:22.036 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-24T23:46:22.036 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-24T23:46:22.036 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-24T23:46:22.036 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-24T23:46:22.036 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-24T23:46:22.036 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-24T23:46:22.036 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-24T23:46:22.036 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T23:46:22.036 INFO:teuthology.orchestra.run.smithi016.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T23:46:22.036 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T23:46:22.037 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-24T23:46:22.037 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-24T23:46:22.037 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-24T23:46:22.037 INFO:teuthology.orchestra.run.smithi016.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T23:46:22.037 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T23:46:22.037 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T23:46:22.037 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T23:46:22.037 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T23:46:22.037 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T23:46:22.037 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-24T23:46:22.037 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-24T23:46:22.037 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-24T23:46:22.037 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-24T23:46:22.037 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-24T23:46:22.038 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-24T23:46:22.038 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-24T23:46:22.038 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T23:46:22.038 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-04-24T23:46:22.038 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-24T23:46:22.038 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-24T23:46:22.038 INFO:teuthology.orchestra.run.smithi016.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T23:46:22.038 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T23:46:22.038 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-04-24T23:46:22.038 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T23:46:22.038 INFO:teuthology.orchestra.run.smithi016.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T23:46:22.038 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T23:46:22.038 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T23:46:22.039 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T23:46:22.040 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-24T23:46:22.040 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T23:46:22.040 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T23:46:22.040 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T23:46:22.040 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T23:46:22.040 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T23:46:22.040 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-24T23:46:22.040 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T23:46:22.040 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-24T23:46:22.040 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T23:46:22.041 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T23:46:22.041 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T23:46:22.041 INFO:teuthology.orchestra.run.smithi016.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T23:46:22.041 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T23:46:22.041 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T23:46:22.041 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T23:46:22.041 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T23:46:22.041 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T23:46:22.041 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T23:46:22.042 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T23:46:22.042 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T23:46:22.042 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T23:46:22.042 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T23:46:22.042 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T23:46:22.042 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T23:46:22.042 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T23:46:22.042 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-24T23:46:22.042 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-24T23:46:22.042 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T23:46:22.042 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T23:46:22.042 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T23:46:22.043 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T23:46:22.043 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T23:46:22.043 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T23:46:22.043 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T23:46:22.043 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T23:46:22.043 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T23:46:22.043 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T23:46:22.043 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T23:46:22.043 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T23:46:22.043 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T23:46:22.043 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T23:46:22.043 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T23:46:22.043 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T23:46:22.043 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T23:46:22.044 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T23:46:22.044 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T23:46:22.044 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T23:46:22.044 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T23:46:22.044 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T23:46:22.044 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T23:46:22.044 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T23:46:22.044 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T23:46:22.044 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T23:46:22.044 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T23:46:22.044 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T23:46:22.044 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T23:46:22.044 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T23:46:22.044 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T23:46:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T23:46:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T23:46:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T23:46:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T23:46:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T23:46:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T23:46:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T23:46:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T23:46:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T23:46:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T23:46:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T23:46:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T23:46:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T23:46:22.045 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T23:46:22.046 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T23:46:22.046 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T23:46:22.046 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T23:46:22.046 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T23:46:22.046 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T23:46:22.046 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T23:46:22.046 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T23:46:22.046 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T23:46:22.046 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T23:46:22.046 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T23:46:22.047 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T23:46:22.047 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T23:46:22.047 INFO:teuthology.orchestra.run.smithi016.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T23:46:22.047 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T23:46:22.047 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T23:46:22.047 INFO:teuthology.orchestra.run.smithi016.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T23:46:22.047 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T23:46:22.047 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T23:46:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-04-24T23:46:22.047 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T23:46:22.047 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:46:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-04-24T23:46:22.047 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================= 2024-04-24T23:46:22.048 INFO:teuthology.orchestra.run.smithi016.stdout:Install 148 Packages 2024-04-24T23:46:22.048 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 4 Packages 2024-04-24T23:46:22.048 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:46:22.048 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 220 M 2024-04-24T23:46:22.048 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-04-24T23:46:22.148 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:16 2024-04-24T23:46:23.014 INFO:teuthology.orchestra.run.smithi016.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-04-24T23:46:23.273 INFO:teuthology.orchestra.run.smithi016.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 3.4 MB/s | 890 kB 00:00 2024-04-24T23:46:23.340 INFO:teuthology.orchestra.run.smithi016.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 2.1 MB/s | 145 kB 00:00 2024-04-24T23:46:23.790 INFO:teuthology.orchestra.run.smithi016.stdout:(4/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 5.2 MB/s | 2.4 MB 00:00 2024-04-24T23:46:23.874 INFO:teuthology.orchestra.run.smithi016.stdout:(5/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 5.2 MB/s | 5.4 MB 00:01 2024-04-24T23:46:24.049 INFO:teuthology.orchestra.run.smithi016.stdout:(6/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 5.8 MB/s | 1.5 MB 00:00 2024-04-24T23:46:24.408 INFO:teuthology.orchestra.run.smithi016.stdout:(7/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 8.8 MB/s | 4.7 MB 00:00 2024-04-24T23:46:25.566 INFO:teuthology.orchestra.run.smithi016.stdout:(8/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el9 8.8 MB/s | 10 MB 00:01 2024-04-24T23:46:25.642 INFO:teuthology.orchestra.run.smithi016.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 335 kB/s | 25 kB 00:00 2024-04-24T23:46:26.017 INFO:teuthology.orchestra.run.smithi016.stdout:(10/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 8.5 MB/s | 17 MB 00:01 2024-04-24T23:46:26.234 INFO:teuthology.orchestra.run.smithi016.stdout:(11/152): ceph-common-19.0.0-2445.gd2c45d97.el9 6.2 MB/s | 21 MB 00:03 2024-04-24T23:46:26.259 INFO:teuthology.orchestra.run.smithi016.stdout:(12/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 920 kB/s | 222 kB 00:00 2024-04-24T23:46:26.284 INFO:teuthology.orchestra.run.smithi016.stdout:(13/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 542 kB/s | 27 kB 00:00 2024-04-24T23:46:26.318 INFO:teuthology.orchestra.run.smithi016.stdout:(14/152): libcephfs-devel-19.0.0-2445.gd2c45d97 540 kB/s | 32 kB 00:00 2024-04-24T23:46:26.402 INFO:teuthology.orchestra.run.smithi016.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 1.9 MB/s | 163 kB 00:00 2024-04-24T23:46:26.444 INFO:teuthology.orchestra.run.smithi016.stdout:(16/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 4.6 MB/s | 745 kB 00:00 2024-04-24T23:46:26.494 INFO:teuthology.orchestra.run.smithi016.stdout:(17/152): librados-devel-19.0.0-2445.gd2c45d97. 1.3 MB/s | 127 kB 00:00 2024-04-24T23:46:26.544 INFO:teuthology.orchestra.run.smithi016.stdout:(18/152): libradosstriper1-19.0.0-2445.gd2c45d9 4.9 MB/s | 501 kB 00:00 2024-04-24T23:46:26.620 INFO:teuthology.orchestra.run.smithi016.stdout:(19/152): python3-ceph-argparse-19.0.0-2445.gd2 602 kB/s | 45 kB 00:00 2024-04-24T23:46:26.703 INFO:teuthology.orchestra.run.smithi016.stdout:(20/152): python3-ceph-common-19.0.0-2445.gd2c4 1.6 MB/s | 133 kB 00:00 2024-04-24T23:46:26.787 INFO:teuthology.orchestra.run.smithi016.stdout:(21/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.9 MB/s | 162 kB 00:00 2024-04-24T23:46:26.879 INFO:teuthology.orchestra.run.smithi016.stdout:(22/152): python3-rados-19.0.0-2445.gd2c45d97.e 3.4 MB/s | 321 kB 00:00 2024-04-24T23:46:26.971 INFO:teuthology.orchestra.run.smithi016.stdout:(23/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 3.2 MB/s | 299 kB 00:00 2024-04-24T23:46:27.046 INFO:teuthology.orchestra.run.smithi016.stdout:(24/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.3 MB/s | 99 kB 00:00 2024-04-24T23:46:27.113 INFO:teuthology.orchestra.run.smithi016.stdout:(25/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.2 MB/s | 85 kB 00:00 2024-04-24T23:46:27.174 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.2 MB 00:01 2024-04-24T23:46:27.680 INFO:teuthology.orchestra.run.smithi016.stdout:(26/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 5.5 MB/s | 3.1 MB 00:00 2024-04-24T23:46:27.755 INFO:teuthology.orchestra.run.smithi016.stdout:(27/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 4.2 MB/s | 5.3 MB 00:01 2024-04-24T23:46:27.780 INFO:teuthology.orchestra.run.smithi016.stdout:(28/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 1.7 MB/s | 171 kB 00:00 2024-04-24T23:46:27.822 INFO:teuthology.orchestra.run.smithi016.stdout:(29/152): ceph-grafana-dashboards-19.0.0-2445.g 475 kB/s | 32 kB 00:00 2024-04-24T23:46:27.856 INFO:teuthology.orchestra.run.smithi016.stdout:(30/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.8 MB/s | 141 kB 00:00 2024-04-24T23:46:28.582 INFO:teuthology.orchestra.run.smithi016.stdout:(31/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 4.6 MB/s | 3.5 MB 00:00 2024-04-24T23:46:28.665 INFO:teuthology.orchestra.run.smithi016.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 2.9 MB/s | 249 kB 00:00 2024-04-24T23:46:28.740 INFO:teuthology.orchestra.run.smithi016.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 660 kB/s | 49 kB 00:00 2024-04-24T23:46:28.816 INFO:teuthology.orchestra.run.smithi016.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 204 kB/s | 15 kB 00:00 2024-04-24T23:46:28.916 INFO:teuthology.orchestra.run.smithi016.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 2.6 MB/s | 264 kB 00:00 2024-04-24T23:46:29.258 INFO:teuthology.orchestra.run.smithi016.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 5.3 MB/s | 7.4 MB 00:01 2024-04-24T23:46:29.283 INFO:teuthology.orchestra.run.smithi016.stdout:(37/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 2.0 MB/s | 762 kB 00:00 2024-04-24T23:46:29.317 INFO:teuthology.orchestra.run.smithi016.stdout:(38/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 312 kB/s | 18 kB 00:00 2024-04-24T23:46:29.483 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-04-24T23:46:29.709 INFO:teuthology.orchestra.run.smithi016.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 189 kB/s | 80 kB 00:00 2024-04-24T23:46:29.984 INFO:teuthology.orchestra.run.smithi016.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 710 kB/s | 195 kB 00:00 2024-04-24T23:46:30.176 INFO:teuthology.orchestra.run.smithi016.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 935 kB/s | 803 kB 00:00 2024-04-24T23:46:30.201 INFO:teuthology.orchestra.run.smithi016.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-04-24T23:46:30.310 INFO:teuthology.orchestra.run.smithi016.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 982 kB/s | 106 kB 00:00 2024-04-24T23:46:30.419 INFO:teuthology.orchestra.run.smithi016.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 135 kB 00:00 2024-04-24T23:46:30.527 INFO:teuthology.orchestra.run.smithi016.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-04-24T23:46:30.636 INFO:teuthology.orchestra.run.smithi016.stdout:(46/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-04-24T23:46:30.761 INFO:teuthology.orchestra.run.smithi016.stdout:(47/152): python3-cryptography-36.0.1-4.el9.x86 2.1 MB/s | 1.2 MB 00:00 2024-04-24T23:46:30.820 INFO:teuthology.orchestra.run.smithi016.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 3.7 MB/s | 689 kB 00:00 2024-04-24T23:46:30.887 INFO:teuthology.orchestra.run.smithi016.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 107 kB 00:00 2024-04-24T23:46:30.929 INFO:teuthology.orchestra.run.smithi016.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 519 kB/s | 21 kB 00:00 2024-04-24T23:46:30.962 INFO:teuthology.orchestra.run.smithi016.stdout:(51/152): bison-3.7.4-5.el9.x86_64.rpm 4.6 MB/s | 943 kB 00:00 2024-04-24T23:46:30.996 INFO:teuthology.orchestra.run.smithi016.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 313 kB 00:00 2024-04-24T23:46:31.021 INFO:teuthology.orchestra.run.smithi016.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-04-24T23:46:31.063 INFO:teuthology.orchestra.run.smithi016.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 437 kB/s | 18 kB 00:00 2024-04-24T23:46:31.122 INFO:teuthology.orchestra.run.smithi016.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-04-24T23:46:31.147 INFO:teuthology.orchestra.run.smithi016.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 133 kB/s | 11 kB 00:00 2024-04-24T23:46:31.172 INFO:teuthology.orchestra.run.smithi016.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-04-24T23:46:31.197 INFO:teuthology.orchestra.run.smithi016.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 903 kB/s | 45 kB 00:00 2024-04-24T23:46:31.239 INFO:teuthology.orchestra.run.smithi016.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-04-24T23:46:31.707 INFO:teuthology.orchestra.run.smithi016.stdout:(60/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 8.0 MB/s | 48 MB 00:06 2024-04-24T23:46:31.723 INFO:teuthology.orchestra.run.smithi016.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 53 kB/s | 25 kB 00:00 2024-04-24T23:46:31.748 INFO:teuthology.orchestra.run.smithi016.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 1.0 MB/s | 582 kB 00:00 2024-04-24T23:46:31.782 INFO:teuthology.orchestra.run.smithi016.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-04-24T23:46:31.808 INFO:teuthology.orchestra.run.smithi016.stdout:(64/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 470 kB/s | 47 kB 00:00 2024-04-24T23:46:31.833 INFO:teuthology.orchestra.run.smithi016.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 697 kB/s | 35 kB 00:00 2024-04-24T23:46:31.858 INFO:teuthology.orchestra.run.smithi016.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-04-24T23:46:31.967 INFO:teuthology.orchestra.run.smithi016.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 33 MB/s | 4.4 MB 00:00 2024-04-24T23:46:31.984 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 21 MB 00:02 2024-04-24T23:46:32.009 INFO:teuthology.orchestra.run.smithi016.stdout:(68/152): postgresql-13.14-1.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-04-24T23:46:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 12 MB/s | 911 kB 00:00 2024-04-24T23:46:32.068 INFO:teuthology.orchestra.run.smithi016.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-24T23:46:32.160 INFO:teuthology.orchestra.run.smithi016.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 13 MB/s | 4.7 MB 00:00 2024-04-24T23:46:32.202 INFO:teuthology.orchestra.run.smithi016.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 7.6 MB/s | 1.0 MB 00:00 2024-04-24T23:46:32.227 INFO:teuthology.orchestra.run.smithi016.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-04-24T23:46:32.319 INFO:teuthology.orchestra.run.smithi016.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 21 MB/s | 5.8 MB 00:00 2024-04-24T23:46:32.344 INFO:teuthology.orchestra.run.smithi016.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.0 MB/s | 244 kB 00:00 2024-04-24T23:46:32.369 INFO:teuthology.orchestra.run.smithi016.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.8 MB/s | 248 kB 00:00 2024-04-24T23:46:32.395 INFO:teuthology.orchestra.run.smithi016.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 509 kB/s | 26 kB 00:00 2024-04-24T23:46:32.420 INFO:teuthology.orchestra.run.smithi016.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-04-24T23:46:32.445 INFO:teuthology.orchestra.run.smithi016.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-24T23:46:32.470 INFO:teuthology.orchestra.run.smithi016.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 693 kB/s | 35 kB 00:00 2024-04-24T23:46:32.513 INFO:teuthology.orchestra.run.smithi016.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.9 MB/s | 250 kB 00:00 2024-04-24T23:46:32.588 INFO:teuthology.orchestra.run.smithi016.stdout:(82/152): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-04-24T23:46:32.613 INFO:teuthology.orchestra.run.smithi016.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-04-24T23:46:32.638 INFO:teuthology.orchestra.run.smithi016.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-04-24T23:46:32.663 INFO:teuthology.orchestra.run.smithi016.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 843 kB/s | 42 kB 00:00 2024-04-24T23:46:32.689 INFO:teuthology.orchestra.run.smithi016.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-24T23:46:32.714 INFO:teuthology.orchestra.run.smithi016.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-24T23:46:32.739 INFO:teuthology.orchestra.run.smithi016.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-24T23:46:32.765 INFO:teuthology.orchestra.run.smithi016.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-24T23:46:32.789 INFO:teuthology.orchestra.run.smithi016.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-24T23:46:32.865 INFO:teuthology.orchestra.run.smithi016.stdout:(91/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-04-24T23:46:32.890 INFO:teuthology.orchestra.run.smithi016.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 415 kB/s | 42 kB 00:00 2024-04-24T23:46:32.916 INFO:teuthology.orchestra.run.smithi016.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 882 kB/s | 44 kB 00:00 2024-04-24T23:46:32.941 INFO:teuthology.orchestra.run.smithi016.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-04-24T23:46:32.968 INFO:teuthology.orchestra.run.smithi016.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 686 kB/s | 36 kB 00:00 2024-04-24T23:46:32.993 INFO:teuthology.orchestra.run.smithi016.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 905 kB/s | 47 kB 00:00 2024-04-24T23:46:33.044 INFO:teuthology.orchestra.run.smithi016.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.0 MB/s | 305 kB 00:00 2024-04-24T23:46:33.069 INFO:teuthology.orchestra.run.smithi016.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 769 kB/s | 58 kB 00:00 2024-04-24T23:46:33.094 INFO:teuthology.orchestra.run.smithi016.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-24T23:46:33.120 INFO:teuthology.orchestra.run.smithi016.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-04-24T23:46:33.145 INFO:teuthology.orchestra.run.smithi016.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 894 kB/s | 45 kB 00:00 2024-04-24T23:46:33.387 INFO:teuthology.orchestra.run.smithi016.stdout:(102/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-04-24T23:46:33.412 INFO:teuthology.orchestra.run.smithi016.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 76 kB/s | 22 kB 00:00 2024-04-24T23:46:33.813 INFO:teuthology.orchestra.run.smithi016.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.3 MB/s | 863 kB 00:00 2024-04-24T23:46:33.863 INFO:teuthology.orchestra.run.smithi016.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 686 kB/s | 309 kB 00:00 2024-04-24T23:46:33.913 INFO:teuthology.orchestra.run.smithi016.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-04-24T23:46:34.005 INFO:teuthology.orchestra.run.smithi016.stdout:(107/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-04-24T23:46:34.038 INFO:teuthology.orchestra.run.smithi016.stdout:(108/152): grpc-data-1.46.7-10.el9.noarch.rpm 86 kB/s | 19 kB 00:00 2024-04-24T23:46:34.089 INFO:teuthology.orchestra.run.smithi016.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 580 kB/s | 49 kB 00:00 2024-04-24T23:46:34.181 INFO:teuthology.orchestra.run.smithi016.stdout:(110/152): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-04-24T23:46:34.206 INFO:teuthology.orchestra.run.smithi016.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 402 kB/s | 67 kB 00:00 2024-04-24T23:46:34.356 INFO:teuthology.orchestra.run.smithi016.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-04-24T23:46:34.432 INFO:teuthology.orchestra.run.smithi016.stdout:(113/152): libarrow-9.0.0-10.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-04-24T23:46:34.457 INFO:teuthology.orchestra.run.smithi016.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 434 kB/s | 43 kB 00:00 2024-04-24T23:46:34.524 INFO:teuthology.orchestra.run.smithi016.stdout:(115/152): python3-asyncssh-2.13.2-5.el9.noarch 1.7 MB/s | 548 kB 00:00 2024-04-24T23:46:34.549 INFO:teuthology.orchestra.run.smithi016.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-04-24T23:46:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-04-24T23:46:34.624 INFO:teuthology.orchestra.run.smithi016.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-04-24T23:46:34.658 INFO:teuthology.orchestra.run.smithi016.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-04-24T23:46:34.683 INFO:teuthology.orchestra.run.smithi016.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-04-24T23:46:34.717 INFO:teuthology.orchestra.run.smithi016.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 383 kB/s | 35 kB 00:00 2024-04-24T23:46:34.750 INFO:teuthology.orchestra.run.smithi016.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.2 MB/s | 206 kB 00:00 2024-04-24T23:46:34.817 INFO:teuthology.orchestra.run.smithi016.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-04-24T23:46:34.843 INFO:teuthology.orchestra.run.smithi016.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-04-24T23:46:34.910 INFO:teuthology.orchestra.run.smithi016.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 191 kB/s | 18 kB 00:00 2024-04-24T23:46:34.935 INFO:teuthology.orchestra.run.smithi016.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-04-24T23:46:35.002 INFO:teuthology.orchestra.run.smithi016.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 212 kB/s | 19 kB 00:00 2024-04-24T23:46:35.027 INFO:teuthology.orchestra.run.smithi016.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-04-24T23:46:35.120 INFO:teuthology.orchestra.run.smithi016.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 504 kB/s | 46 kB 00:00 2024-04-24T23:46:35.186 INFO:teuthology.orchestra.run.smithi016.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 5.6 MB/s | 1.0 MB 00:00 2024-04-24T23:46:35.212 INFO:teuthology.orchestra.run.smithi016.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 859 kB/s | 79 kB 00:00 2024-04-24T23:46:35.279 INFO:teuthology.orchestra.run.smithi016.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 626 kB/s | 58 kB 00:00 2024-04-24T23:46:35.320 INFO:teuthology.orchestra.run.smithi016.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-24T23:46:35.371 INFO:teuthology.orchestra.run.smithi016.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2024-04-24T23:46:35.412 INFO:teuthology.orchestra.run.smithi016.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 981 kB/s | 90 kB 00:00 2024-04-24T23:46:35.471 INFO:teuthology.orchestra.run.smithi016.stdout:(136/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.6 MB/s | 2.0 MB 00:00 2024-04-24T23:46:35.497 INFO:teuthology.orchestra.run.smithi016.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 225 kB/s | 28 kB 00:00 2024-04-24T23:46:35.522 INFO:teuthology.orchestra.run.smithi016.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 282 kB/s | 31 kB 00:00 2024-04-24T23:46:35.572 INFO:teuthology.orchestra.run.smithi016.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-24T23:46:35.597 INFO:teuthology.orchestra.run.smithi016.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 589 kB/s | 59 kB 00:00 2024-04-24T23:46:35.622 INFO:teuthology.orchestra.run.smithi016.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-04-24T23:46:35.664 INFO:teuthology.orchestra.run.smithi016.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 553 kB/s | 51 kB 00:00 2024-04-24T23:46:35.699 INFO:teuthology.orchestra.run.smithi016.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-04-24T23:46:35.723 INFO:teuthology.orchestra.run.smithi016.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 890 kB/s | 90 kB 00:00 2024-04-24T23:46:35.765 INFO:teuthology.orchestra.run.smithi016.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-04-24T23:46:35.790 INFO:teuthology.orchestra.run.smithi016.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-04-24T23:46:35.816 INFO:teuthology.orchestra.run.smithi016.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-04-24T23:46:35.891 INFO:teuthology.orchestra.run.smithi016.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-24T23:46:36.575 INFO:teuthology.orchestra.run.smithi016.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 2.0 MB/s | 1.3 MB 00:00 2024-04-24T23:46:36.800 INFO:teuthology.orchestra.run.smithi016.stdout:(150/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 3.2 MB/s | 3.1 MB 00:00 2024-04-24T23:46:36.917 INFO:teuthology.orchestra.run.smithi016.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 7.0 MB/s | 2.4 MB 00:00 2024-04-24T23:46:36.976 INFO:teuthology.orchestra.run.smithi016.stdout:(152/152): librados2-19.0.0-2445.gd2c45d97.el9. 2.9 MB/s | 3.4 MB 00:01 2024-04-24T23:46:36.978 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-04-24T23:46:36.979 INFO:teuthology.orchestra.run.smithi016.stdout:Total 15 MB/s | 220 MB 00:14 2024-04-24T23:46:37.118 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-04-24T23:46:38.104 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-04-24T23:46:38.265 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-04-24T23:46:38.265 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-04-24T23:46:39.252 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T23:46:39.253 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T23:46:39.264 INFO:teuthology.orchestra.run.smithi156.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T23:46:39.264 INFO:teuthology.orchestra.run.smithi156.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T23:46:39.265 INFO:teuthology.orchestra.run.smithi156.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T23:46:39.266 INFO:teuthology.orchestra.run.smithi156.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T23:46:39.267 INFO:teuthology.orchestra.run.smithi156.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T23:46:39.270 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T23:46:39.271 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T23:46:39.330 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-04-24T23:46:39.339 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================= 2024-04-24T23:46:39.339 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================= 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 6.6 k 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.4 M 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 890 k 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 145 k 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 1.5 M 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 141 k 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 3.5 M 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 7.4 M 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 49 k 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 10 M 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 48 M 2024-04-24T23:46:39.340 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 264 k 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 762 k 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 222 k 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 27 k 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 18 k 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 32 k 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 745 k 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 127 k 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T23:46:39.341 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T23:46:39.342 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T23:46:39.342 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 162 k 2024-04-24T23:46:39.342 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 321 k 2024-04-24T23:46:39.342 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 299 k 2024-04-24T23:46:39.342 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 99 k 2024-04-24T23:46:39.342 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 85 k 2024-04-24T23:46:39.342 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-24T23:46:39.342 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 171 k 2024-04-24T23:46:39.342 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T23:46:39.342 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2024-04-24T23:46:39.342 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.4 M 2024-04-24T23:46:39.342 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 3.1 M 2024-04-24T23:46:39.342 INFO:teuthology.orchestra.run.smithi156.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T23:46:39.342 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 21 M 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 32 k 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 2.4 M 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 249 k 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 4.7 M 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 17 M 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2445.gd2c45d97.el9 ceph-noarch 15 k 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 25 k 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T23:46:39.343 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 163 k 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 501 k 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 5.3 M 2024-04-24T23:46:39.344 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T23:46:39.345 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T23:46:39.345 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T23:46:39.345 INFO:teuthology.orchestra.run.smithi156.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T23:46:39.345 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T23:46:39.345 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T23:46:39.345 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T23:46:39.345 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T23:46:39.345 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T23:46:39.345 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T23:46:39.345 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T23:46:39.345 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T23:46:39.345 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T23:46:39.346 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T23:46:39.346 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T23:46:39.346 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T23:46:39.346 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T23:46:39.346 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 45 k 2024-04-24T23:46:39.346 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:19.0.0-2445.gd2c45d97.el9 ceph 133 k 2024-04-24T23:46:39.346 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T23:46:39.346 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T23:46:39.346 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T23:46:39.346 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T23:46:39.346 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T23:46:39.346 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T23:46:39.346 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T23:46:39.346 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T23:46:39.347 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T23:46:39.348 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T23:46:39.349 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T23:46:39.349 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T23:46:39.349 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T23:46:39.349 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T23:46:39.349 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T23:46:39.349 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T23:46:39.349 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T23:46:39.349 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T23:46:39.349 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T23:46:39.349 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T23:46:39.349 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T23:46:39.349 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T23:46:39.349 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T23:46:39.349 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T23:46:39.350 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T23:46:39.350 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T23:46:39.350 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T23:46:39.350 INFO:teuthology.orchestra.run.smithi156.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T23:46:39.350 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T23:46:39.350 INFO:teuthology.orchestra.run.smithi156.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T23:46:39.350 INFO:teuthology.orchestra.run.smithi156.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T23:46:39.350 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T23:46:39.350 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T23:46:39.350 INFO:teuthology.orchestra.run.smithi156.stdout:Installing weak dependencies: 2024-04-24T23:46:39.350 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T23:46:39.350 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:46:39.350 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-04-24T23:46:39.350 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================= 2024-04-24T23:46:39.351 INFO:teuthology.orchestra.run.smithi156.stdout:Install 148 Packages 2024-04-24T23:46:39.351 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 4 Packages 2024-04-24T23:46:39.351 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:46:39.351 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 220 M 2024-04-24T23:46:39.351 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-04-24T23:46:40.102 INFO:teuthology.orchestra.run.smithi156.stdout:(1/152): ceph-19.0.0-2445.gd2c45d97.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-04-24T23:46:40.126 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-04-24T23:46:40.126 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-04-24T23:46:40.944 INFO:teuthology.orchestra.run.smithi156.stdout:(2/152): ceph-fuse-19.0.0-2445.gd2c45d97.el9.x8 1.0 MB/s | 890 kB 00:00 2024-04-24T23:46:41.103 INFO:teuthology.orchestra.run.smithi156.stdout:(3/152): ceph-immutable-object-cache-19.0.0-244 916 kB/s | 145 kB 00:00 2024-04-24T23:46:41.836 INFO:teuthology.orchestra.run.smithi156.stdout:(4/152): ceph-base-19.0.0-2445.gd2c45d97.el9.x8 2.6 MB/s | 5.4 MB 00:02 2024-04-24T23:46:42.120 INFO:teuthology.orchestra.run.smithi156.stdout:(5/152): ceph-mds-19.0.0-2445.gd2c45d97.el9.x86 2.3 MB/s | 2.4 MB 00:01 2024-04-24T23:46:42.153 INFO:teuthology.orchestra.run.smithi156.stdout:(6/152): ceph-mgr-19.0.0-2445.gd2c45d97.el9.x86 4.7 MB/s | 1.5 MB 00:00 2024-04-24T23:46:43.754 INFO:teuthology.orchestra.run.smithi156.stdout:(7/152): ceph-mon-19.0.0-2445.gd2c45d97.el9.x86 2.9 MB/s | 4.7 MB 00:01 2024-04-24T23:46:43.996 INFO:teuthology.orchestra.run.smithi156.stdout:(8/152): ceph-common-19.0.0-2445.gd2c45d97.el9. 5.0 MB/s | 21 MB 00:04 2024-04-24T23:46:44.062 INFO:teuthology.orchestra.run.smithi156.stdout:(9/152): ceph-selinux-19.0.0-2445.gd2c45d97.el9 379 kB/s | 25 kB 00:00 2024-04-24T23:46:44.813 INFO:teuthology.orchestra.run.smithi156.stdout:(10/152): ceph-osd-19.0.0-2445.gd2c45d97.el9.x8 6.3 MB/s | 17 MB 00:02 2024-04-24T23:46:44.929 INFO:teuthology.orchestra.run.smithi156.stdout:(11/152): cephfs-mirror-19.0.0-2445.gd2c45d97.e 1.9 MB/s | 222 kB 00:00 2024-04-24T23:46:45.013 INFO:teuthology.orchestra.run.smithi156.stdout:(12/152): cephfs-shell-19.0.0-2445.gd2c45d97.el 325 kB/s | 27 kB 00:00 2024-04-24T23:46:45.088 INFO:teuthology.orchestra.run.smithi156.stdout:(13/152): libcephfs-devel-19.0.0-2445.gd2c45d97 423 kB/s | 32 kB 00:00 2024-04-24T23:46:45.264 INFO:teuthology.orchestra.run.smithi156.stdout:(14/152): libcephfs2-19.0.0-2445.gd2c45d97.el9. 4.2 MB/s | 745 kB 00:00 2024-04-24T23:46:45.331 INFO:teuthology.orchestra.run.smithi156.stdout:(15/152): libcephsqlite-19.0.0-2445.gd2c45d97.e 2.4 MB/s | 163 kB 00:00 2024-04-24T23:46:45.414 INFO:teuthology.orchestra.run.smithi156.stdout:(16/152): librados-devel-19.0.0-2445.gd2c45d97. 1.5 MB/s | 127 kB 00:00 2024-04-24T23:46:45.573 INFO:teuthology.orchestra.run.smithi156.stdout:(17/152): libradosstriper1-19.0.0-2445.gd2c45d9 3.1 MB/s | 501 kB 00:00 2024-04-24T23:46:46.415 INFO:teuthology.orchestra.run.smithi156.stdout:(18/152): librgw2-19.0.0-2445.gd2c45d97.el9.x86 6.3 MB/s | 5.3 MB 00:00 2024-04-24T23:46:46.511 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-04-24T23:46:46.573 INFO:teuthology.orchestra.run.smithi156.stdout:(19/152): ceph-radosgw-19.0.0-2445.gd2c45d97.el 3.6 MB/s | 10 MB 00:02 2024-04-24T23:46:46.596 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T23:46:46.598 INFO:teuthology.orchestra.run.smithi156.stdout:(20/152): python3-ceph-argparse-19.0.0-2445.gd2 246 kB/s | 45 kB 00:00 2024-04-24T23:46:46.632 INFO:teuthology.orchestra.run.smithi156.stdout:(21/152): python3-ceph-common-19.0.0-2445.gd2c4 2.2 MB/s | 133 kB 00:00 2024-04-24T23:46:46.635 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T23:46:46.689 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T23:46:46.691 INFO:teuthology.orchestra.run.smithi156.stdout:(22/152): python3-cephfs-19.0.0-2445.gd2c45d97. 1.7 MB/s | 162 kB 00:00 2024-04-24T23:46:46.739 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T23:46:46.764 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T23:46:46.766 INFO:teuthology.orchestra.run.smithi156.stdout:(23/152): python3-rados-19.0.0-2445.gd2c45d97.e 2.3 MB/s | 321 kB 00:00 2024-04-24T23:46:46.800 INFO:teuthology.orchestra.run.smithi156.stdout:(24/152): python3-rbd-19.0.0-2445.gd2c45d97.el9 2.7 MB/s | 299 kB 00:00 2024-04-24T23:46:46.808 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T23:46:46.842 INFO:teuthology.orchestra.run.smithi156.stdout:(25/152): python3-rgw-19.0.0-2445.gd2c45d97.el9 1.3 MB/s | 99 kB 00:00 2024-04-24T23:46:46.959 INFO:teuthology.orchestra.run.smithi156.stdout:(26/152): rbd-fuse-19.0.0-2445.gd2c45d97.el9.x8 537 kB/s | 85 kB 00:00 2024-04-24T23:46:47.051 INFO:teuthology.orchestra.run.smithi156.stdout:(27/152): rbd-nbd-19.0.0-2445.gd2c45d97.el9.x86 1.8 MB/s | 171 kB 00:00 2024-04-24T23:46:47.151 INFO:teuthology.orchestra.run.smithi156.stdout:(28/152): ceph-grafana-dashboards-19.0.0-2445.g 317 kB/s | 32 kB 00:00 2024-04-24T23:46:47.158 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T23:46:47.252 INFO:teuthology.orchestra.run.smithi156.stdout:(29/152): ceph-mgr-cephadm-19.0.0-2445.gd2c45d9 1.4 MB/s | 141 kB 00:00 2024-04-24T23:46:47.265 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:46:47.382 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:46:47.408 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:46:47.484 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:46:47.512 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-24T23:46:47.669 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T23:46:47.718 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T23:46:47.745 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:46:47.851 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:46:47.880 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-24T23:46:47.943 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-24T23:46:47.964 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T23:46:47.983 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T23:46:48.035 INFO:teuthology.orchestra.run.smithi156.stdout:(30/152): ceph-mgr-dashboard-19.0.0-2445.gd2c45 4.5 MB/s | 3.5 MB 00:00 2024-04-24T23:46:48.182 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T23:46:48.214 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T23:46:48.325 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T23:46:48.427 INFO:teuthology.orchestra.run.smithi156.stdout:(31/152): rbd-mirror-19.0.0-2445.gd2c45d97.el9. 1.9 MB/s | 3.1 MB 00:01 2024-04-24T23:46:48.460 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T23:46:48.497 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T23:46:48.557 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T23:46:48.594 INFO:teuthology.orchestra.run.smithi156.stdout:(32/152): ceph-mgr-modules-core-19.0.0-2445.gd2 1.5 MB/s | 249 kB 00:00 2024-04-24T23:46:48.631 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T23:46:48.672 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T23:46:48.677 INFO:teuthology.orchestra.run.smithi156.stdout:(33/152): ceph-mgr-rook-19.0.0-2445.gd2c45d97.e 593 kB/s | 49 kB 00:00 2024-04-24T23:46:48.716 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T23:46:48.748 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T23:46:48.772 INFO:teuthology.orchestra.run.smithi156.stdout:(34/152): ceph-prometheus-alerts-19.0.0-2445.gd 163 kB/s | 15 kB 00:00 2024-04-24T23:46:48.776 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T23:46:48.839 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T23:46:48.889 INFO:teuthology.orchestra.run.smithi156.stdout:(35/152): ceph-volume-19.0.0-2445.gd2c45d97.el9 2.2 MB/s | 264 kB 00:00 2024-04-24T23:46:48.891 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T23:46:48.920 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T23:46:48.964 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T23:46:49.042 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T23:46:49.084 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-24T23:46:49.128 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-24T23:46:49.155 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-24T23:46:49.172 INFO:teuthology.orchestra.run.smithi156.stdout:(36/152): cephadm-19.0.0-2445.gd2c45d97.el9.noa 2.6 MB/s | 762 kB 00:00 2024-04-24T23:46:49.230 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T23:46:49.287 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T23:46:49.306 INFO:teuthology.orchestra.run.smithi156.stdout:(37/152): ceph-mgr-diskprediction-local-19.0.0- 5.8 MB/s | 7.4 MB 00:01 2024-04-24T23:46:49.331 INFO:teuthology.orchestra.run.smithi156.stdout:(38/152): cephfs-top-19.0.0-2445.gd2c45d97.el9. 115 kB/s | 18 kB 00:00 2024-04-24T23:46:49.365 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T23:46:49.440 INFO:teuthology.orchestra.run.smithi156.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 602 kB/s | 80 kB 00:00 2024-04-24T23:46:49.507 INFO:teuthology.orchestra.run.smithi156.stdout:(40/152): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-04-24T23:46:49.509 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T23:46:49.532 INFO:teuthology.orchestra.run.smithi156.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-24T23:46:49.557 INFO:teuthology.orchestra.run.smithi156.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.9 MB/s | 253 kB 00:00 2024-04-24T23:46:49.565 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T23:46:49.599 INFO:teuthology.orchestra.run.smithi156.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-04-24T23:46:49.641 INFO:teuthology.orchestra.run.smithi156.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-04-24T23:46:49.667 INFO:teuthology.orchestra.run.smithi156.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-04-24T23:46:49.692 INFO:teuthology.orchestra.run.smithi156.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-04-24T23:46:49.717 INFO:teuthology.orchestra.run.smithi156.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-04-24T23:46:49.801 INFO:teuthology.orchestra.run.smithi156.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 6.2 MB/s | 689 kB 00:00 2024-04-24T23:46:49.843 INFO:teuthology.orchestra.run.smithi156.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 7.4 MB/s | 943 kB 00:00 2024-04-24T23:46:49.885 INFO:teuthology.orchestra.run.smithi156.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 1.2 MB/s | 107 kB 00:00 2024-04-24T23:46:49.910 INFO:teuthology.orchestra.run.smithi156.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 321 kB/s | 21 kB 00:00 2024-04-24T23:46:49.960 INFO:teuthology.orchestra.run.smithi156.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 657 kB/s | 33 kB 00:00 2024-04-24T23:46:49.985 INFO:teuthology.orchestra.run.smithi156.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 3.1 MB/s | 313 kB 00:00 2024-04-24T23:46:50.036 INFO:teuthology.orchestra.run.smithi156.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 365 kB/s | 18 kB 00:00 2024-04-24T23:46:50.132 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T23:46:50.220 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T23:46:50.264 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T23:46:50.307 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T23:46:50.345 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-24T23:46:50.387 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T23:46:50.461 INFO:teuthology.orchestra.run.smithi156.stdout:(55/152): ceph-test-19.0.0-2445.gd2c45d97.el9.x 7.5 MB/s | 48 MB 00:06 2024-04-24T23:46:50.495 INFO:teuthology.orchestra.run.smithi156.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 24 kB/s | 11 kB 00:00 2024-04-24T23:46:50.537 INFO:teuthology.orchestra.run.smithi156.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-24T23:46:50.553 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T23:46:50.562 INFO:teuthology.orchestra.run.smithi156.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 678 kB/s | 45 kB 00:00 2024-04-24T23:46:50.612 INFO:teuthology.orchestra.run.smithi156.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 4.6 MB/s | 3.0 MB 00:00 2024-04-24T23:46:50.614 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T23:46:50.662 INFO:teuthology.orchestra.run.smithi156.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 512 kB/s | 25 kB 00:00 2024-04-24T23:46:50.671 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T23:46:50.688 INFO:teuthology.orchestra.run.smithi156.stdout:(61/152): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-04-24T23:46:50.713 INFO:teuthology.orchestra.run.smithi156.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 3.8 MB/s | 582 kB 00:00 2024-04-24T23:46:50.730 INFO:teuthology.orchestra.run.smithi156.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 704 kB/s | 47 kB 00:00 2024-04-24T23:46:50.745 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T23:46:50.772 INFO:teuthology.orchestra.run.smithi156.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 848 kB/s | 35 kB 00:00 2024-04-24T23:46:50.781 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T23:46:50.797 INFO:teuthology.orchestra.run.smithi156.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-24T23:46:50.818 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T23:46:50.822 INFO:teuthology.orchestra.run.smithi156.stdout:(66/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-04-24T23:46:50.866 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T23:46:50.901 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T23:46:50.945 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T23:46:50.987 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T23:46:51.014 INFO:teuthology.orchestra.run.smithi156.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 8.3 MB/s | 1.6 MB 00:00 2024-04-24T23:46:51.106 INFO:teuthology.orchestra.run.smithi156.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 14 MB/s | 4.7 MB 00:00 2024-04-24T23:46:51.131 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T23:46:51.173 INFO:teuthology.orchestra.run.smithi156.stdout:(69/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-04-24T23:46:51.184 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T23:46:51.213 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T23:46:51.232 INFO:teuthology.orchestra.run.smithi156.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.0 MB/s | 133 kB 00:00 2024-04-24T23:46:51.255 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T23:46:51.257 INFO:teuthology.orchestra.run.smithi156.stdout:(71/152): postgresql-contrib-13.14-1.el9.x86_64 3.7 MB/s | 911 kB 00:00 2024-04-24T23:46:51.349 INFO:teuthology.orchestra.run.smithi156.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 954 kB/s | 87 kB 00:00 2024-04-24T23:46:51.355 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T23:46:51.383 INFO:teuthology.orchestra.run.smithi156.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-04-24T23:46:51.389 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T23:46:51.429 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T23:46:51.462 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T23:46:51.487 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T23:46:51.516 INFO:teuthology.orchestra.run.smithi156.stdout:(74/152): postgresql-server-13.14-1.el9.x86_64. 17 MB/s | 5.8 MB 00:00 2024-04-24T23:46:51.541 INFO:teuthology.orchestra.run.smithi156.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.5 MB/s | 244 kB 00:00 2024-04-24T23:46:51.567 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T23:46:51.592 INFO:teuthology.orchestra.run.smithi156.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 512 kB/s | 26 kB 00:00 2024-04-24T23:46:51.617 INFO:teuthology.orchestra.run.smithi156.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-04-24T23:46:51.659 INFO:teuthology.orchestra.run.smithi156.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 286 kB/s | 19 kB 00:00 2024-04-24T23:46:51.703 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T23:46:51.743 INFO:teuthology.orchestra.run.smithi156.stdout:(79/152): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-04-24T23:46:51.744 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T23:46:51.768 INFO:teuthology.orchestra.run.smithi156.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-24T23:46:51.772 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T23:46:51.793 INFO:teuthology.orchestra.run.smithi156.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 260 kB/s | 35 kB 00:00 2024-04-24T23:46:51.807 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T23:46:51.868 INFO:teuthology.orchestra.run.smithi156.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-04-24T23:46:51.894 INFO:teuthology.orchestra.run.smithi156.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 2.2 MB/s | 222 kB 00:00 2024-04-24T23:46:51.930 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T23:46:51.952 INFO:teuthology.orchestra.run.smithi156.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 922 kB/s | 77 kB 00:00 2024-04-24T23:46:51.978 INFO:teuthology.orchestra.run.smithi156.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 507 kB/s | 42 kB 00:00 2024-04-24T23:46:52.053 INFO:teuthology.orchestra.run.smithi156.stdout:(86/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 5.1 MB 00:00 2024-04-24T23:46:52.095 INFO:teuthology.orchestra.run.smithi156.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-04-24T23:46:52.120 INFO:teuthology.orchestra.run.smithi156.stdout:(88/152): python3-protobuf-3.14.0-13.el9.noarch 1.6 MB/s | 269 kB 00:00 2024-04-24T23:46:52.146 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T23:46:52.162 INFO:teuthology.orchestra.run.smithi156.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-04-24T23:46:52.175 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T23:46:52.187 INFO:teuthology.orchestra.run.smithi156.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 554 kB/s | 51 kB 00:00 2024-04-24T23:46:52.210 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T23:46:52.212 INFO:teuthology.orchestra.run.smithi156.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 582 kB/s | 54 kB 00:00 2024-04-24T23:46:52.263 INFO:teuthology.orchestra.run.smithi156.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-04-24T23:46:52.268 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T23:46:52.288 INFO:teuthology.orchestra.run.smithi156.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 584 kB/s | 44 kB 00:00 2024-04-24T23:46:52.311 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T23:46:52.338 INFO:teuthology.orchestra.run.smithi156.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 718 kB/s | 36 kB 00:00 2024-04-24T23:46:52.344 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T23:46:52.364 INFO:teuthology.orchestra.run.smithi156.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.1 MB/s | 221 kB 00:00 2024-04-24T23:46:52.414 INFO:teuthology.orchestra.run.smithi156.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 629 kB/s | 47 kB 00:00 2024-04-24T23:46:52.440 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T23:46:52.464 INFO:teuthology.orchestra.run.smithi156.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.0 MB/s | 305 kB 00:00 2024-04-24T23:46:52.484 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T23:46:52.490 INFO:teuthology.orchestra.run.smithi156.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 767 kB/s | 58 kB 00:00 2024-04-24T23:46:52.522 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T23:46:52.557 INFO:teuthology.orchestra.run.smithi156.stdout:(99/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 955 kB/s | 64 kB 00:00 2024-04-24T23:46:52.570 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T23:46:52.582 INFO:teuthology.orchestra.run.smithi156.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 798 kB/s | 94 kB 00:00 2024-04-24T23:46:52.670 INFO:teuthology.orchestra.run.smithi156.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 594 kB/s | 45 kB 00:00 2024-04-24T23:46:52.670 INFO:teuthology.orchestra.run.smithi156.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 334 kB/s | 22 kB 00:00 2024-04-24T23:46:52.690 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T23:46:52.749 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T23:46:52.808 INFO:teuthology.orchestra.run.smithi156.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-04-24T23:46:52.875 INFO:teuthology.orchestra.run.smithi156.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 3.5 MB/s | 863 kB 00:00 2024-04-24T23:46:52.909 INFO:teuthology.orchestra.run.smithi156.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 3.0 MB/s | 309 kB 00:00 2024-04-24T23:46:52.934 INFO:teuthology.orchestra.run.smithi156.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.9 MB/s | 548 kB 00:00 2024-04-24T23:46:52.950 INFO:teuthology.orchestra.run.smithi156.stdout:(107/152): grpc-data-1.46.7-10.el9.noarch.rpm 257 kB/s | 19 kB 00:00 2024-04-24T23:46:52.968 INFO:teuthology.orchestra.run.smithi156.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 777 kB/s | 26 kB 00:00 2024-04-24T23:46:52.993 INFO:teuthology.orchestra.run.smithi156.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-04-24T23:46:53.018 INFO:teuthology.orchestra.run.smithi156.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-24T23:46:53.085 INFO:teuthology.orchestra.run.smithi156.stdout:(111/152): libarrow-9.0.0-10.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2024-04-24T23:46:53.110 INFO:teuthology.orchestra.run.smithi156.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-04-24T23:46:53.136 INFO:teuthology.orchestra.run.smithi156.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 11 MB/s | 548 kB 00:00 2024-04-24T23:46:53.161 INFO:teuthology.orchestra.run.smithi156.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 866 kB/s | 43 kB 00:00 2024-04-24T23:46:53.186 INFO:teuthology.orchestra.run.smithi156.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 642 kB/s | 32 kB 00:00 2024-04-24T23:46:53.211 INFO:teuthology.orchestra.run.smithi156.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-04-24T23:46:53.237 INFO:teuthology.orchestra.run.smithi156.stdout:(117/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.7 MB/s | 838 kB 00:00 2024-04-24T23:46:53.262 INFO:teuthology.orchestra.run.smithi156.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-04-24T23:46:53.287 INFO:teuthology.orchestra.run.smithi156.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-04-24T23:46:53.321 INFO:teuthology.orchestra.run.smithi156.stdout:(120/152): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-04-24T23:46:53.346 INFO:teuthology.orchestra.run.smithi156.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 422 kB/s | 35 kB 00:00 2024-04-24T23:46:53.371 INFO:teuthology.orchestra.run.smithi156.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.4 MB/s | 206 kB 00:00 2024-04-24T23:46:53.397 INFO:teuthology.orchestra.run.smithi156.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-04-24T23:46:53.422 INFO:teuthology.orchestra.run.smithi156.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 212 kB/s | 11 kB 00:00 2024-04-24T23:46:53.447 INFO:teuthology.orchestra.run.smithi156.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 351 kB/s | 18 kB 00:00 2024-04-24T23:46:53.472 INFO:teuthology.orchestra.run.smithi156.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-04-24T23:46:53.498 INFO:teuthology.orchestra.run.smithi156.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 388 kB/s | 19 kB 00:00 2024-04-24T23:46:53.513 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T23:46:53.523 INFO:teuthology.orchestra.run.smithi156.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 390 kB/s | 20 kB 00:00 2024-04-24T23:46:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T23:46:53.573 INFO:teuthology.orchestra.run.smithi156.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 9.0 MB/s | 2.0 MB 00:00 2024-04-24T23:46:53.590 INFO:teuthology.orchestra.run.smithi156.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 692 kB/s | 46 kB 00:00 2024-04-24T23:46:53.595 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T23:46:53.616 INFO:teuthology.orchestra.run.smithi156.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 1.8 MB/s | 79 kB 00:00 2024-04-24T23:46:53.633 INFO:teuthology.orchestra.run.smithi156.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 1.3 MB/s | 58 kB 00:00 2024-04-24T23:46:53.666 INFO:teuthology.orchestra.run.smithi156.stdout:(133/152): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-04-24T23:46:53.692 INFO:teuthology.orchestra.run.smithi156.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-04-24T23:46:53.698 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T23:46:53.717 INFO:teuthology.orchestra.run.smithi156.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-04-24T23:46:53.742 INFO:teuthology.orchestra.run.smithi156.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 564 kB/s | 28 kB 00:00 2024-04-24T23:46:53.754 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T23:46:53.767 INFO:teuthology.orchestra.run.smithi156.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-04-24T23:46:53.792 INFO:teuthology.orchestra.run.smithi156.stdout:(138/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 714 kB/s | 90 kB 00:00 2024-04-24T23:46:53.818 INFO:teuthology.orchestra.run.smithi156.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-04-24T23:46:53.843 INFO:teuthology.orchestra.run.smithi156.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 710 kB/s | 36 kB 00:00 2024-04-24T23:46:53.845 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T23:46:53.868 INFO:teuthology.orchestra.run.smithi156.stdout:(141/152): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-04-24T23:46:53.894 INFO:teuthology.orchestra.run.smithi156.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 674 kB/s | 51 kB 00:00 2024-04-24T23:46:53.917 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T23:46:53.927 INFO:teuthology.orchestra.run.smithi156.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-24T23:46:53.953 INFO:teuthology.orchestra.run.smithi156.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.5 MB/s | 90 kB 00:00 2024-04-24T23:46:53.978 INFO:teuthology.orchestra.run.smithi156.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-04-24T23:46:54.011 INFO:teuthology.orchestra.run.smithi156.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-24T23:46:54.036 INFO:teuthology.orchestra.run.smithi156.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-04-24T23:46:54.079 INFO:teuthology.orchestra.run.smithi156.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-24T23:46:54.214 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T23:46:54.237 INFO:teuthology.orchestra.run.smithi156.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 8.5 MB/s | 1.3 MB 00:00 2024-04-24T23:46:54.245 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T23:46:54.313 INFO:teuthology.orchestra.run.smithi156.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 32 MB/s | 2.4 MB 00:00 2024-04-24T23:46:54.325 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T23:46:54.421 INFO:teuthology.orchestra.run.smithi156.stdout:(151/152): librbd1-19.0.0-2445.gd2c45d97.el9.x8 8.2 MB/s | 3.1 MB 00:00 2024-04-24T23:46:54.696 INFO:teuthology.orchestra.run.smithi156.stdout:(152/152): librados2-19.0.0-2445.gd2c45d97.el9. 4.9 MB/s | 3.4 MB 00:00 2024-04-24T23:46:54.699 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-04-24T23:46:54.700 INFO:teuthology.orchestra.run.smithi156.stdout:Total 14 MB/s | 220 MB 00:15 2024-04-24T23:46:55.008 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T23:46:55.122 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T23:46:55.804 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-04-24T23:46:55.945 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-04-24T23:46:55.945 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-04-24T23:46:56.435 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T23:46:56.507 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T23:46:56.536 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T23:46:56.569 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T23:46:56.889 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T23:46:56.939 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-24T23:46:57.015 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-24T23:46:57.044 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-24T23:46:57.081 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T23:46:57.579 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T23:46:57.609 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-24T23:46:57.681 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-24T23:46:57.697 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-04-24T23:46:57.697 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-04-24T23:46:57.711 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-24T23:47:00.084 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:47:00.116 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:47:00.687 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:47:00.736 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-24T23:47:00.810 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-24T23:47:00.810 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T23:47:00.811 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T23:47:00.811 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:00.845 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-24T23:47:03.633 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-04-24T23:47:03.734 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T23:47:03.772 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T23:47:03.834 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T23:47:03.868 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T23:47:03.901 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T23:47:03.944 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T23:47:04.275 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T23:47:04.411 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:47:04.592 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:47:04.621 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:47:04.696 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:47:04.733 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-24T23:47:04.884 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T23:47:04.931 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T23:47:04.958 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:47:05.062 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:47:05.084 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-24T23:47:05.147 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 14/156 2024-04-24T23:47:05.177 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T23:47:05.201 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T23:47:05.394 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T23:47:05.435 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T23:47:05.543 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T23:47:05.645 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T23:47:05.684 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T23:47:05.719 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T23:47:05.784 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T23:47:05.818 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T23:47:05.862 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T23:47:05.894 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T23:47:05.922 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T23:47:05.991 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T23:47:06.044 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T23:47:06.084 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T23:47:06.119 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T23:47:06.195 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T23:47:06.229 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 31/156 2024-04-24T23:47:06.273 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 32/156 2024-04-24T23:47:06.302 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 33/156 2024-04-24T23:47:06.371 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T23:47:06.425 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T23:47:06.499 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T23:47:06.632 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T23:47:06.694 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T23:47:07.231 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T23:47:07.315 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T23:47:07.344 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T23:47:07.386 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T23:47:07.425 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 43/156 2024-04-24T23:47:07.466 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T23:47:07.625 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T23:47:07.686 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T23:47:07.740 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T23:47:07.813 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T23:47:07.845 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T23:47:07.881 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T23:47:07.928 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T23:47:07.964 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T23:47:08.007 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T23:47:08.049 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T23:47:08.186 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T23:47:08.229 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T23:47:08.260 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T23:47:08.301 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T23:47:08.395 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T23:47:08.435 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T23:47:08.475 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T23:47:08.509 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T23:47:08.533 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T23:47:08.608 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T23:47:08.737 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T23:47:08.774 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T23:47:08.810 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T23:47:08.845 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T23:47:08.884 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T23:47:08.916 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T23:47:08.946 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T23:47:08.972 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T23:47:09.005 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T23:47:09.047 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T23:47:09.081 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T23:47:09.168 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T23:47:09.212 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T23:47:09.268 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T23:47:09.323 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T23:47:09.422 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T23:47:09.470 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T23:47:10.188 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T23:47:10.242 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T23:47:10.275 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T23:47:10.375 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T23:47:10.433 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T23:47:10.520 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T23:47:10.552 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T23:47:10.731 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T23:47:10.759 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T23:47:10.837 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T23:47:11.486 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T23:47:11.598 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T23:47:12.823 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T23:47:12.832 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-24T23:47:12.832 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-04-24T23:47:12.832 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-04-24T23:47:12.832 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-04-24T23:47:12.832 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-04-24T23:47:12.832 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-04-24T23:47:12.832 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /root 2024-04-24T23:47:12.832 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-04-24T23:47:12.832 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:12.905 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T23:47:12.933 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T23:47:12.966 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T23:47:13.008 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-24T23:47:13.024 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-24T23:47:13.024 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:13.024 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T23:47:13.024 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:13.279 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T23:47:13.495 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-24T23:47:13.694 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 99/156 2024-04-24T23:47:13.717 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 100/156 2024-04-24T23:47:13.762 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T23:47:14.229 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T23:47:14.257 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-24T23:47:14.311 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 103/156 2024-04-24T23:47:14.342 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 104/156 2024-04-24T23:47:14.569 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-24T23:47:14.596 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-24T23:47:14.596 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:14.596 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T23:47:14.597 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T23:47:14.597 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T23:47:14.597 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:14.837 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-24T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-24T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T23:47:14.861 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:15.329 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-24T23:47:15.353 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-24T23:47:15.353 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:15.354 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T23:47:15.354 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T23:47:15.354 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T23:47:15.354 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:15.460 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T23:47:15.518 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T23:47:15.582 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T23:47:15.786 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T23:47:15.830 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T23:47:15.911 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T23:47:16.003 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T23:47:16.153 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T23:47:16.197 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T23:47:16.229 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T23:47:16.256 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T23:47:16.624 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:47:16.663 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:47:16.715 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-24T23:47:16.746 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-24T23:47:16.778 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-24T23:47:16.842 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-24T23:47:16.869 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-24T23:47:17.197 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 105/156 2024-04-24T23:47:17.242 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-24T23:47:17.334 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 106/156 2024-04-24T23:47:17.334 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T23:47:17.334 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T23:47:17.334 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:17.368 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-24T23:47:17.964 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-24T23:47:18.004 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-24T23:47:19.024 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-24T23:47:19.059 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-24T23:47:19.168 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-24T23:47:19.334 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-24T23:47:19.365 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-24T23:47:19.390 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-24T23:47:19.390 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:19.390 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T23:47:19.390 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T23:47:19.390 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T23:47:19.390 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:19.442 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-24T23:47:19.452 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-24T23:47:19.555 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-24T23:47:19.607 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-24T23:47:20.555 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T23:47:20.585 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-24T23:47:20.609 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-24T23:47:20.610 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:20.610 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T23:47:20.610 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:47:20.610 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:47:20.610 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:20.653 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-24T23:47:20.675 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-24T23:47:20.675 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:20.676 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T23:47:20.676 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:20.728 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-24T23:47:20.752 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-24T23:47:20.752 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:20.752 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T23:47:20.752 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:47:20.752 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:47:20.753 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:21.080 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-24T23:47:21.104 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-24T23:47:21.104 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:21.105 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T23:47:21.105 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T23:47:21.105 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T23:47:21.105 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:25.875 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-24T23:47:25.916 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-24T23:47:25.958 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-24T23:47:25.989 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T23:47:26.016 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T23:47:26.213 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-24T23:47:26.254 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T23:47:26.283 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-24T23:47:26.878 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:47:26.931 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:47:26.951 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:47:27.849 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-24T23:47:27.944 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T23:47:27.986 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T23:47:28.070 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T23:47:28.098 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T23:47:28.099 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T23:47:28.174 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T23:47:28.215 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T23:47:28.215 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T23:47:28.306 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T23:47:28.375 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:47:28.851 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 107/156 2024-04-24T23:47:28.851 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2024-04-24T23:47:28.851 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2024-04-24T23:47:28.851 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2024-04-24T23:47:28.851 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2024-04-24T23:47:28.851 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2024-04-24T23:47:28.851 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /root 2024-04-24T23:47:28.851 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2024-04-24T23:47:28.851 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:29.005 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-24T23:47:29.021 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 108/156 2024-04-24T23:47:29.021 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:29.021 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T23:47:29.021 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:30.483 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-24T23:47:30.509 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 109/156 2024-04-24T23:47:30.509 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:30.509 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T23:47:30.509 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T23:47:30.509 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T23:47:30.509 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:30.743 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-24T23:47:30.766 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 110/156 2024-04-24T23:47:30.766 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:30.766 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T23:47:30.766 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T23:47:30.766 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T23:47:30.766 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:31.219 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-24T23:47:31.242 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 111/156 2024-04-24T23:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T23:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T23:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T23:47:31.243 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:31.350 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T23:47:31.407 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T23:47:31.462 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T23:47:31.648 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T23:47:31.695 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T23:47:31.774 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T23:47:31.863 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T23:47:32.003 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T23:47:32.045 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T23:47:32.077 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T23:47:32.114 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T23:47:32.546 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-24T23:47:32.562 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 123/156 2024-04-24T23:47:32.585 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 124/156 2024-04-24T23:47:32.647 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 125/156 2024-04-24T23:47:32.677 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-24T23:47:33.713 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 126/156 2024-04-24T23:47:33.762 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-24T23:47:34.037 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:47:34.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-24T23:47:34.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-24T23:47:34.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-24T23:47:34.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-24T23:47:34.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-24T23:47:34.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-24T23:47:34.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-24T23:47:34.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:47:34.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:47:34.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-24T23:47:34.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-24T23:47:34.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-24T23:47:34.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:47:34.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-24T23:47:34.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-24T23:47:34.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-24T23:47:34.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-24T23:47:34.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-24T23:47:34.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-24T23:47:34.038 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-24T23:47:34.039 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-24T23:47:34.039 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-24T23:47:34.039 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-24T23:47:34.039 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-24T23:47:34.039 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-24T23:47:34.039 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-24T23:47:34.039 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-24T23:47:34.051 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T23:47:34.052 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T23:47:34.053 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T23:47:34.054 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T23:47:34.054 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T23:47:34.054 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T23:47:34.054 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T23:47:34.054 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T23:47:34.054 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T23:47:34.054 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T23:47:34.054 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T23:47:34.054 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T23:47:34.054 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T23:47:34.054 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T23:47:34.054 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T23:47:34.054 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T23:47:34.054 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T23:47:34.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T23:47:34.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T23:47:34.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T23:47:34.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T23:47:34.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T23:47:34.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T23:47:34.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T23:47:34.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T23:47:34.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T23:47:34.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T23:47:34.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T23:47:34.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T23:47:34.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T23:47:34.055 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T23:47:34.056 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T23:47:34.057 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T23:47:34.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T23:47:34.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T23:47:34.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T23:47:34.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T23:47:34.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T23:47:34.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T23:47:34.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T23:47:34.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T23:47:34.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T23:47:34.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T23:47:34.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T23:47:34.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T23:47:34.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T23:47:34.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-24T23:47:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T23:47:34.060 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T23:47:34.733 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 127/156 2024-04-24T23:47:34.762 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-24T23:47:34.865 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 128/156 2024-04-24T23:47:35.026 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 129/156 2024-04-24T23:47:35.064 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-24T23:47:35.088 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 130/156 2024-04-24T23:47:35.088 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:35.088 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T23:47:35.088 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T23:47:35.088 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T23:47:35.088 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:35.132 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-24T23:47:35.143 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 131/156 2024-04-24T23:47:35.233 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 132/156 2024-04-24T23:47:35.281 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 133/156 2024-04-24T23:47:36.181 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T23:47:36.209 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-24T23:47:36.232 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 135/156 2024-04-24T23:47:36.232 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:36.232 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T23:47:36.232 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:47:36.232 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T23:47:36.232 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:36.262 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:47:36.262 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:36.263 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-04-24T23:47:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T23:47:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T23:47:36.263 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-04-24T23:47:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T23:47:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T23:47:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T23:47:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T23:47:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.263 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.264 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T23:47:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T23:47:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T23:47:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T23:47:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T23:47:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T23:47:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T23:47:36.265 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T23:47:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T23:47:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T23:47:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T23:47:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T23:47:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T23:47:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T23:47:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T23:47:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T23:47:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.266 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T23:47:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T23:47:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T23:47:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T23:47:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T23:47:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T23:47:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T23:47:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T23:47:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T23:47:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T23:47:36.267 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.268 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T23:47:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T23:47:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T23:47:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T23:47:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T23:47:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T23:47:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T23:47:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T23:47:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T23:47:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T23:47:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T23:47:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T23:47:36.269 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T23:47:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T23:47:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T23:47:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T23:47:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T23:47:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T23:47:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T23:47:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T23:47:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T23:47:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T23:47:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T23:47:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T23:47:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T23:47:36.270 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T23:47:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T23:47:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T23:47:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T23:47:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T23:47:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T23:47:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T23:47:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T23:47:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T23:47:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T23:47:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T23:47:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T23:47:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T23:47:36.271 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T23:47:36.272 INFO:teuthology.orchestra.run.smithi016.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T23:47:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T23:47:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T23:47:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T23:47:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T23:47:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T23:47:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T23:47:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T23:47:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T23:47:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T23:47:36.273 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T23:47:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T23:47:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T23:47:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T23:47:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T23:47:36.274 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:36.274 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-04-24T23:47:36.276 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-24T23:47:36.299 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 136/156 2024-04-24T23:47:36.299 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:36.299 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T23:47:36.299 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:36.344 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-24T23:47:36.366 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 137/156 2024-04-24T23:47:36.367 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:36.367 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T23:47:36.367 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:47:36.367 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T23:47:36.367 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:36.659 DEBUG:teuthology.parallel:result is None 2024-04-24T23:47:36.670 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-24T23:47:36.692 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 138/156 2024-04-24T23:47:36.693 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T23:47:36.693 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T23:47:36.693 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T23:47:36.693 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T23:47:36.693 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:41.190 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 139/156 2024-04-24T23:47:41.232 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 140/156 2024-04-24T23:47:41.272 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 141/156 2024-04-24T23:47:41.305 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T23:47:41.332 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T23:47:41.520 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 144/156 2024-04-24T23:47:41.562 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T23:47:41.599 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 146/156 2024-04-24T23:47:42.249 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:47:42.289 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:47:42.309 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T23:47:43.155 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 148/156 2024-04-24T23:47:43.236 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T23:47:43.276 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T23:47:43.365 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T23:47:43.407 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T23:47:43.407 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T23:47:43.491 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T23:47:43.524 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T23:47:43.524 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T23:47:43.622 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T23:47:43.689 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:47:49.747 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:47:49.747 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 1/156 2024-04-24T23:47:49.747 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2/156 2024-04-24T23:47:49.747 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 3/156 2024-04-24T23:47:49.747 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 4/156 2024-04-24T23:47:49.747 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2445.gd2c45 5/156 2024-04-24T23:47:49.748 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 6/156 2024-04-24T23:47:49.748 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 7/156 2024-04-24T23:47:49.748 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 8/156 2024-04-24T23:47:49.748 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 9/156 2024-04-24T23:47:49.748 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 10/156 2024-04-24T23:47:49.748 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 11/156 2024-04-24T23:47:49.748 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 12/156 2024-04-24T23:47:49.748 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 13/156 2024-04-24T23:47:49.748 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 14/156 2024-04-24T23:47:49.748 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_ 15/156 2024-04-24T23:47:49.748 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 16/156 2024-04-24T23:47:49.748 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 17/156 2024-04-24T23:47:49.748 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_6 18/156 2024-04-24T23:47:49.748 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86 19/156 2024-04-24T23:47:49.749 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 20/156 2024-04-24T23:47:49.749 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el 21/156 2024-04-24T23:47:49.749 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9. 22/156 2024-04-24T23:47:49.749 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_6 23/156 2024-04-24T23:47:49.749 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 24/156 2024-04-24T23:47:49.749 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 25/156 2024-04-24T23:47:49.749 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 26/156 2024-04-24T23:47:49.749 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 27/156 2024-04-24T23:47:49.761 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 28/156 2024-04-24T23:47:49.761 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 29/156 2024-04-24T23:47:49.761 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97. 30/156 2024-04-24T23:47:49.761 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noa 31/156 2024-04-24T23:47:49.761 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.n 32/156 2024-04-24T23:47:49.761 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c 33/156 2024-04-24T23:47:49.761 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el 34/156 2024-04-24T23:47:49.762 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 35/156 2024-04-24T23:47:49.762 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.e 36/156 2024-04-24T23:47:49.762 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 37/156 2024-04-24T23:47:49.762 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 38/156 2024-04-24T23:47:49.762 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 39/156 2024-04-24T23:47:49.762 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T23:47:49.762 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T23:47:49.762 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T23:47:49.762 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T23:47:49.762 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T23:47:49.762 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T23:47:49.762 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T23:47:49.762 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T23:47:49.763 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T23:47:49.763 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T23:47:49.763 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T23:47:49.763 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T23:47:49.763 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T23:47:49.763 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T23:47:49.763 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T23:47:49.763 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T23:47:49.763 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T23:47:49.763 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T23:47:49.763 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T23:47:49.763 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T23:47:49.763 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T23:47:49.763 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T23:47:49.764 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T23:47:49.764 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T23:47:49.764 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T23:47:49.764 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T23:47:49.764 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T23:47:49.764 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T23:47:49.764 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T23:47:49.764 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T23:47:49.764 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T23:47:49.764 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T23:47:49.764 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T23:47:49.764 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T23:47:49.764 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T23:47:49.765 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T23:47:49.765 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T23:47:49.765 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T23:47:49.765 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T23:47:49.765 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T23:47:49.765 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T23:47:49.765 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T23:47:49.765 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T23:47:49.765 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T23:47:49.765 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T23:47:49.765 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T23:47:49.765 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T23:47:49.765 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T23:47:49.765 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T23:47:49.766 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T23:47:49.766 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T23:47:49.766 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T23:47:49.766 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T23:47:49.766 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T23:47:49.766 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T23:47:49.766 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T23:47:49.766 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T23:47:49.766 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T23:47:49.766 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T23:47:49.766 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T23:47:49.766 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T23:47:49.766 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T23:47:49.767 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T23:47:49.768 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T23:47:49.769 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T23:47:49.770 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T23:47:49.770 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 149/156 2024-04-24T23:47:49.770 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T23:47:49.770 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 151/156 2024-04-24T23:47:49.770 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T23:47:49.770 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T23:47:49.770 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T23:47:49.770 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T23:47:51.833 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T23:47:51.833 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:51.834 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2024-04-24T23:47:51.834 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.834 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.834 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T23:47:51.834 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T23:47:51.834 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-04-24T23:47:51.834 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T23:47:51.834 INFO:teuthology.orchestra.run.smithi156.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T23:47:51.834 INFO:teuthology.orchestra.run.smithi156.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T23:47:51.834 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T23:47:51.834 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.834 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.834 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.835 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.835 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:51.835 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.835 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.835 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.835 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:51.835 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:51.835 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:51.835 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:51.835 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:51.835 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.835 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.835 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:51.836 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.836 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.836 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.836 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:51.836 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:51.836 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.836 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.836 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top-2:19.0.0-2445.gd2c45d97.el9.noarch 2024-04-24T23:47:51.836 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T23:47:51.836 INFO:teuthology.orchestra.run.smithi156.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T23:47:51.836 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T23:47:51.836 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T23:47:51.836 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T23:47:51.837 INFO:teuthology.orchestra.run.smithi156.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T23:47:51.837 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T23:47:51.837 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T23:47:51.837 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T23:47:51.837 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T23:47:51.837 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T23:47:51.837 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.837 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.837 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.837 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T23:47:51.837 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T23:47:51.837 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T23:47:51.837 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T23:47:51.837 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T23:47:51.838 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.838 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.838 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T23:47:51.838 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.838 INFO:teuthology.orchestra.run.smithi156.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T23:47:51.838 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T23:47:51.838 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T23:47:51.838 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T23:47:51.838 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T23:47:51.838 INFO:teuthology.orchestra.run.smithi156.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T23:47:51.838 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T23:47:51.838 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T23:47:51.838 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T23:47:51.839 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T23:47:51.839 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T23:47:51.839 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T23:47:51.839 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T23:47:51.839 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T23:47:51.839 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T23:47:51.839 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T23:47:51.839 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T23:47:51.839 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T23:47:51.839 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T23:47:51.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T23:47:51.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T23:47:51.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T23:47:51.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T23:47:51.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T23:47:51.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T23:47:51.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T23:47:51.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T23:47:51.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T23:47:51.840 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T23:47:51.841 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T23:47:51.841 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T23:47:51.841 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T23:47:51.841 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T23:47:51.841 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T23:47:51.841 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T23:47:51.841 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T23:47:51.841 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T23:47:51.841 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T23:47:51.841 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T23:47:51.841 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T23:47:51.841 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T23:47:51.841 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T23:47:51.842 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T23:47:51.842 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T23:47:51.842 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T23:47:51.842 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T23:47:51.842 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T23:47:51.842 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T23:47:51.842 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T23:47:51.842 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T23:47:51.842 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T23:47:51.842 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T23:47:51.842 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T23:47:51.842 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T23:47:51.842 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T23:47:51.843 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T23:47:51.843 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T23:47:51.843 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T23:47:51.843 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T23:47:51.843 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T23:47:51.843 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T23:47:51.843 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T23:47:51.843 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T23:47:51.843 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.843 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.843 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T23:47:51.843 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T23:47:51.843 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T23:47:51.844 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.844 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T23:47:51.844 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T23:47:51.844 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T23:47:51.844 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T23:47:51.844 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T23:47:51.844 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T23:47:51.844 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T23:47:51.844 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T23:47:51.844 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T23:47:51.844 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T23:47:51.844 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T23:47:51.844 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T23:47:51.844 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T23:47:51.845 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T23:47:51.845 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T23:47:51.845 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.845 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.845 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:19.0.0-2445.gd2c45d97.el9.x86_64 2024-04-24T23:47:51.845 INFO:teuthology.orchestra.run.smithi156.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T23:47:51.845 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T23:47:51.845 INFO:teuthology.orchestra.run.smithi156.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T23:47:51.845 INFO:teuthology.orchestra.run.smithi156.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T23:47:51.845 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T23:47:51.845 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T23:47:51.846 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T23:47:51.846 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:51.846 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-04-24T23:47:52.235 DEBUG:teuthology.parallel:result is None 2024-04-24T23:47:52.235 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:47:52.375 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T23:47:52.402 INFO:teuthology.orchestra.run.smithi016.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-24T23:47:52.403 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-24T23:47:52.403 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-24T23:47:52.405 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d2c45d977142e024a59b68a8f1dce8f0b9840551 2024-04-24T23:47:52.539 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T23:47:52.560 INFO:teuthology.orchestra.run.smithi156.stdout:19.0.0-2445.gd2c45d97.el9 2024-04-24T23:47:52.560 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2445.gd2c45d97.el9 2024-04-24T23:47:52.560 INFO:teuthology.task.install:The correct ceph version 19.0.0-2445.gd2c45d97 is installed. 2024-04-24T23:47:52.562 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-24T23:47:52.562 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:47:52.562 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T23:47:52.595 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:47:52.595 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T23:47:52.626 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-24T23:47:52.626 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:47:52.626 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T23:47:52.665 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T23:47:52.733 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:47:52.734 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T23:47:52.757 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T23:47:52.820 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-24T23:47:52.821 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:47:52.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T23:47:52.848 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T23:47:52.915 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:47:52.915 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T23:47:52.939 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T23:47:53.004 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-24T23:47:53.004 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:47:53.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T23:47:53.031 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T23:47:53.097 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:47:53.097 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T23:47:53.122 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T23:47:53.185 INFO:teuthology.run_tasks:Running task ceph... 2024-04-24T23:47:53.268 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-04-24T23:47:53.268 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /var/log/ceph 2024-04-24T23:47:53.271 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /var/log/ceph 2024-04-24T23:47:53.295 INFO:tasks.ceph:Disabling ceph logrotate... 2024-04-24T23:47:53.295 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T23:47:53.337 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/logrotate.d/ceph 2024-04-24T23:47:53.361 INFO:tasks.ceph:Creating extra log directories... 2024-04-24T23:47:53.361 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T23:47:53.403 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-04-24T23:47:53.435 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-04-24T23:47:53.435 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-04-24T23:47:53.435 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-04-24T23:47:53.435 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:47:53.435 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-24T23:47:53.482 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-24T23:47:53.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-24T23:47:53.615 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:47:53.615 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-04-24T23:47:53.631 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-04-24T23:47:53.741 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-04-24T23:47:53.784 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-04-24T23:47:53.784 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-04-24T23:47:53.784 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219', 'branch': 'squid', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/recovery-fs}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7672219', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 8528, 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'sleep_before_teardown': 0, 'subset': '122/512', 'suite': 'fs', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_d2c45d977142e024a59b68a8f1dce8f0b9840551/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi016.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYsIB19cMcfrT6kPr3H+ZQvTnJ3B601Az23vEK1M8i2MEhX932Bvugfhcgl2rvWXqSCW79GONnaiWvJvY6YON0=', 'smithi156.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMnVq9hvlZNZZI3TLOcEDPIl6FQneeSABfDPoVkLzqN/MwqPBV2bVTfvrQtQTERyEMFWiEALYEF4zj68ooehLOo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': 'd2c45d977142e024a59b68a8f1dce8f0b9840551', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_recovery_fs']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17', 'timestamp': '2024-04-24_21:24:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-04-24T23:47:53.785 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T23:47:53.787 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-04-24T23:47:53.833 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T23:47:53.843 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m0777 -- /var/run/ceph 2024-04-24T23:47:53.896 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:47:53.897 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T23:47:53.920 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T23:47:53.920 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-04-24T23:47:53.975 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T23:47:53.976 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:47:53.976 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-24T23:47:53.976 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:47:53.976 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:47:53.976 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-24 23:47:28.763035541 +0000 2024-04-24T23:47:53.976 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-24 23:45:42.275457418 +0000 2024-04-24T23:47:53.976 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-24 23:45:42.275457418 +0000 2024-04-24T23:47:53.976 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-24 23:45:42.275457418 +0000 2024-04-24T23:47:53.976 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T23:47:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-24T23:47:54.038 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-24T23:47:54.038 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 9.9781e-05 s, 5.1 MB/s 2024-04-24T23:47:54.039 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T23:47:54.095 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-04-24T23:47:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T23:47:54.150 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:47:54.150 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-24T23:47:54.150 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:47:54.150 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:47:54.150 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-24 23:47:28.764035524 +0000 2024-04-24T23:47:54.151 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-24 23:45:42.621451834 +0000 2024-04-24T23:47:54.151 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-24 23:45:42.621451834 +0000 2024-04-24T23:47:54.151 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-24 23:45:42.621451834 +0000 2024-04-24T23:47:54.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T23:47:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-24T23:47:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-24T23:47:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 9.5841e-05 s, 5.3 MB/s 2024-04-24T23:47:54.214 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T23:47:54.270 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-04-24T23:47:54.326 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T23:47:54.326 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:47:54.326 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-24T23:47:54.326 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:47:54.326 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:47:54.326 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-24 23:47:28.764035524 +0000 2024-04-24T23:47:54.326 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-24 23:45:42.984445976 +0000 2024-04-24T23:47:54.326 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-24 23:45:42.984445976 +0000 2024-04-24T23:47:54.326 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-24 23:45:42.984445976 +0000 2024-04-24T23:47:54.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T23:47:54.389 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-24T23:47:54.389 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-24T23:47:54.389 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 9.9035e-05 s, 5.2 MB/s 2024-04-24T23:47:54.390 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T23:47:54.446 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-04-24T23:47:54.501 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T23:47:54.502 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:47:54.502 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-24T23:47:54.502 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:47:54.502 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:47:54.502 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-04-24 23:47:28.765035507 +0000 2024-04-24T23:47:54.502 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-04-24 23:45:43.315440633 +0000 2024-04-24T23:47:54.502 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-04-24 23:45:43.315440633 +0000 2024-04-24T23:47:54.502 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-04-24 23:45:43.315440633 +0000 2024-04-24T23:47:54.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T23:47:54.564 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-04-24T23:47:54.564 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-04-24T23:47:54.564 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 8.5278e-05 s, 6.0 MB/s 2024-04-24T23:47:54.566 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T23:47:54.622 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T23:47:54.622 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:47:54.622 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T23:47:54.638 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T23:47:54.638 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2024-04-24T23:47:54.693 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T23:47:54.693 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:47:54.693 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-24T23:47:54.693 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:47:54.693 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:47:54.693 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-24 23:47:44.066943399 +0000 2024-04-24T23:47:54.694 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-24 23:45:39.482729590 +0000 2024-04-24T23:47:54.694 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-24 23:45:39.482729590 +0000 2024-04-24T23:47:54.694 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-24 23:45:39.482729590 +0000 2024-04-24T23:47:54.694 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T23:47:54.755 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-24T23:47:54.755 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-24T23:47:54.755 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 9.2189e-05 s, 5.6 MB/s 2024-04-24T23:47:54.756 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T23:47:54.811 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2024-04-24T23:47:54.867 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T23:47:54.867 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:47:54.867 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-24T23:47:54.867 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:47:54.867 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:47:54.867 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-24 23:47:44.066943399 +0000 2024-04-24T23:47:54.868 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-24 23:45:39.804724929 +0000 2024-04-24T23:47:54.868 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-24 23:45:39.804724929 +0000 2024-04-24T23:47:54.868 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-24 23:45:39.804724929 +0000 2024-04-24T23:47:54.868 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T23:47:54.929 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-24T23:47:54.929 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-24T23:47:54.929 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 8.59e-05 s, 6.0 MB/s 2024-04-24T23:47:54.930 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T23:47:54.986 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2024-04-24T23:47:55.040 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T23:47:55.040 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:47:55.040 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-04-24T23:47:55.040 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:47:55.041 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:47:55.041 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-24 23:47:44.066943399 +0000 2024-04-24T23:47:55.041 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-24 23:45:40.151719909 +0000 2024-04-24T23:47:55.041 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-24 23:45:40.151719909 +0000 2024-04-24T23:47:55.041 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-24 23:45:40.151719909 +0000 2024-04-24T23:47:55.041 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T23:47:55.102 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-24T23:47:55.102 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-24T23:47:55.102 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000107481 s, 4.8 MB/s 2024-04-24T23:47:55.104 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T23:47:55.159 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2024-04-24T23:47:55.214 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T23:47:55.214 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T23:47:55.214 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-24T23:47:55.214 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T23:47:55.214 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T23:47:55.214 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-04-24 23:47:44.067943384 +0000 2024-04-24T23:47:55.214 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-04-24 23:45:40.479715164 +0000 2024-04-24T23:47:55.214 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-04-24 23:45:40.479715164 +0000 2024-04-24T23:47:55.214 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-04-24 23:45:40.479715164 +0000 2024-04-24T23:47:55.215 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T23:47:55.276 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-04-24T23:47:55.277 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-04-24T23:47:55.277 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000105028 s, 4.9 MB/s 2024-04-24T23:47:55.278 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T23:47:55.333 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T23:47:55.334 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-24T23:47:55.334 INFO:tasks.ceph:Generating config... 2024-04-24T23:47:55.335 INFO:tasks.ceph:[client] client mount timeout = 600 2024-04-24T23:47:55.335 INFO:tasks.ceph:[client] debug client = 20 2024-04-24T23:47:55.335 INFO:tasks.ceph:[client] debug ms = 1 2024-04-24T23:47:55.335 INFO:tasks.ceph:[client] pid file = 2024-04-24T23:47:55.335 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-04-24T23:47:55.335 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-04-24T23:47:55.335 INFO:tasks.ceph:[mds] debug mds = 20 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mds] debug ms = 1 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mds] mds debug frag = True 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mgr] debug client = 20 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mgr] debug ms = 1 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mon] debug mon = 20 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mon] debug ms = 1 2024-04-24T23:47:55.336 INFO:tasks.ceph:[mon] debug paxos = 20 2024-04-24T23:47:55.337 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-04-24T23:47:55.337 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-04-24T23:47:55.337 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-04-24T23:47:55.337 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-04-24T23:47:55.337 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-04-24T23:47:55.337 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-04-24T23:47:55.337 INFO:tasks.ceph:[osd] debug ms = 1 2024-04-24T23:47:55.337 INFO:tasks.ceph:[osd] debug osd = 20 2024-04-24T23:47:55.337 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-04-24T23:47:55.337 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-04-24T23:47:55.337 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-04-24T23:47:55.337 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-04-24T23:47:55.337 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-04-24T23:47:55.338 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-04-24T23:47:55.338 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-04-24T23:47:55.338 INFO:tasks.ceph:Setting up mon.a... 2024-04-24T23:47:55.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-04-24T23:47:55.391 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.keyring 2024-04-24T23:47:55.394 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-04-24T23:47:55.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T23:47:55.465 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.16'), ('mon.b', '172.21.15.156'), ('mon.c', '[v2:172.21.15.156:3301,v1:172.21.15.156:6790]')] 2024-04-24T23:47:55.466 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.16,172.21.15.156,[v2:172.21.15.156:3301,v1:172.21.15.156:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-04-24T23:47:55.466 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:47:55.467 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T23:47:55.521 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.16 --add b 172.21.15.156 --addv c '[v2:172.21.15.156:3301,v1:172.21.15.156:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:47:55.607 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:47:55.608 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: generated fsid f8999c8c-ab57-4aba-8d53-08e19e6ee3d5 2024-04-24T23:47:55.608 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = quincy 2024-04-24T23:47:55.608 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 0 2024-04-24T23:47:55.608 INFO:teuthology.orchestra.run.smithi016.stdout:fsid f8999c8c-ab57-4aba-8d53-08e19e6ee3d5 2024-04-24T23:47:55.608 INFO:teuthology.orchestra.run.smithi016.stdout:last_changed 2024-04-24T23:47:55.609169+0000 2024-04-24T23:47:55.608 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-04-24T23:47:55.609169+0000 2024-04-24T23:47:55.608 INFO:teuthology.orchestra.run.smithi016.stdout:min_mon_release 17 (quincy) 2024-04-24T23:47:55.608 INFO:teuthology.orchestra.run.smithi016.stdout:election_strategy: 1 2024-04-24T23:47:55.608 INFO:teuthology.orchestra.run.smithi016.stdout:0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-04-24T23:47:55.608 INFO:teuthology.orchestra.run.smithi016.stdout:1: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.b 2024-04-24T23:47:55.608 INFO:teuthology.orchestra.run.smithi016.stdout:2: [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] mon.c 2024-04-24T23:47:55.608 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-04-24T23:47:55.609 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-04-24T23:47:55.624 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f8999c8c-ab57-4aba-8d53-08e19e6ee3d5... 2024-04-24T23:47:55.625 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T23:47:55.667 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-04-24T23:47:55.706 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-04-24T23:47:55.706 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2024-04-24T23:47:55.707 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T23:47:55.708 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2024-04-24T23:47:55.709 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.709 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T23:47:55.709 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.709 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2024-04-24T23:47:55.709 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.709 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2024-04-24T23:47:55.709 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.709 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2024-04-24T23:47:55.709 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2024-04-24T23:47:55.709 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2024-04-24T23:47:55.709 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = "172.21.15.16,172.21.15.156,[v2:172.21.15.156:3301,v1:172.21.15.156:6790]" 2024-04-24T23:47:55.709 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = f8999c8c-ab57-4aba-8d53-08e19e6ee3d5 2024-04-24T23:47:55.709 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.709 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2024-04-24T23:47:55.710 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2024-04-24T23:47:55.710 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.710 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2024-04-24T23:47:55.710 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2024-04-24T23:47:55.710 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock profile = high_recovery_ops 2024-04-24T23:47:55.710 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.710 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2024-04-24T23:47:55.710 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2024-04-24T23:47:55.710 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.710 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2024-04-24T23:47:55.710 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2024-04-24T23:47:55.710 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2024-04-24T23:47:55.710 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug trim objects = true 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2024-04-24T23:47:55.711 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2024-04-24T23:47:55.712 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2024-04-24T23:47:55.712 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2024-04-24T23:47:55.712 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 20 2024-04-24T23:47:55.712 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 20 2024-04-24T23:47:55.712 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-04-24T23:47:55.712 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2024-04-24T23:47:55.712 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 10 2024-04-24T23:47:55.712 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T23:47:55.712 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2024-04-24T23:47:55.712 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2024-04-24T23:47:55.712 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2024-04-24T23:47:55.712 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2024-04-24T23:47:55.712 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2024-04-24T23:47:55.712 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.713 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2024-04-24T23:47:55.713 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-04-24T23:47:55.713 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2024-04-24T23:47:55.713 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-04-24T23:47:55.713 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-04-24T23:47:55.713 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-04-24T23:47:55.713 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-04-24T23:47:55.713 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2024-04-24T23:47:55.713 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2024-04-24T23:47:55.713 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.713 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2024-04-24T23:47:55.713 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-04-24T23:47:55.713 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-04-24T23:47:55.713 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T23:47:55.714 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T23:47:55.715 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.715 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2024-04-24T23:47:55.715 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2024-04-24T23:47:55.715 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.715 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2024-04-24T23:47:55.715 INFO:teuthology.orchestra.run.smithi016.stdout: mon down mkfs grace = 300 2024-04-24T23:47:55.715 INFO:teuthology.orchestra.run.smithi016.stdout: mon op complaint time = 120 2024-04-24T23:47:55.715 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:47:55.715 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2024-04-24T23:47:55.715 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2024-04-24T23:47:55.715 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2024-04-24T23:47:55.715 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2024-04-24T23:47:55.715 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout: client mount timeout = 600 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout: debug client = 20 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = "" 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.b] 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.c] 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a] 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c] 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b] 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d] 2024-04-24T23:47:55.716 INFO:teuthology.orchestra.run.smithi016.stdout:[mds] 2024-04-24T23:47:55.717 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds = 20 2024-04-24T23:47:55.717 INFO:teuthology.orchestra.run.smithi016.stdout: debug mds balancer = 20 2024-04-24T23:47:55.717 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-04-24T23:47:55.717 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug frag = True 2024-04-24T23:47:55.717 INFO:teuthology.orchestra.run.smithi016.stdout: mds debug scatterstat = True 2024-04-24T23:47:55.717 INFO:teuthology.orchestra.run.smithi016.stdout: mds op complaint time = 180 2024-04-24T23:47:55.717 INFO:teuthology.orchestra.run.smithi016.stdout: mds verify scatter = True 2024-04-24T23:47:55.717 INFO:teuthology.orchestra.run.smithi016.stdout: osd op complaint time = 180 2024-04-24T23:47:55.717 INFO:teuthology.orchestra.run.smithi016.stdout: rados mon op timeout = 900 2024-04-24T23:47:55.717 INFO:teuthology.orchestra.run.smithi016.stdout: rados osd op timeout = 900 2024-04-24T23:47:55.718 INFO:teuthology.orchestra.run.smithi156.stdout:[global] 2024-04-24T23:47:55.718 INFO:teuthology.orchestra.run.smithi156.stdout: chdir = "" 2024-04-24T23:47:55.718 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-04-24T23:47:55.718 INFO:teuthology.orchestra.run.smithi156.stdout: auth supported = cephx 2024-04-24T23:47:55.718 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.718 INFO:teuthology.orchestra.run.smithi156.stdout: filestore xattr use omap = true 2024-04-24T23:47:55.718 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.718 INFO:teuthology.orchestra.run.smithi156.stdout: mon clock drift allowed = 1.000 2024-04-24T23:47:55.718 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.718 INFO:teuthology.orchestra.run.smithi156.stdout: osd crush chooseleaf type = 0 2024-04-24T23:47:55.718 INFO:teuthology.orchestra.run.smithi156.stdout: auth debug = true 2024-04-24T23:47:55.718 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.719 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on old message = true 2024-04-24T23:47:55.719 INFO:teuthology.orchestra.run.smithi156.stdout: ms die on bug = true 2024-04-24T23:47:55.719 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.719 INFO:teuthology.orchestra.run.smithi156.stdout: mon max pg per osd = 10000 # >= luminous 2024-04-24T23:47:55.719 INFO:teuthology.orchestra.run.smithi156.stdout: mon pg warn max object skew = 0 2024-04-24T23:47:55.719 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.719 INFO:teuthology.orchestra.run.smithi156.stdout: # disable pg_autoscaler by default for new pools 2024-04-24T23:47:55.719 INFO:teuthology.orchestra.run.smithi156.stdout: osd_pool_default_pg_autoscale_mode = off 2024-04-24T23:47:55.719 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.719 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default size = 2 2024-04-24T23:47:55.719 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.719 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow primary affinity = true 2024-04-24T23:47:55.719 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd allow pg remap = true 2024-04-24T23:47:55.719 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on legacy crush tunables = false 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on crush straw calc version zero = false 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on no sortbitwise = false 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on osd down out interval zero = false 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: mon warn on too few osds = false 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_pool_no_redundancy = false 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: mon_allow_pool_size_one = true 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: osd default data pool replay window = 5 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: mon allow pool delete = true 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: mon cluster log file level = debug 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: debug asserts on shutdown = true 2024-04-24T23:47:55.720 INFO:teuthology.orchestra.run.smithi156.stdout: mon health detail to clog = false 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: mon host = "172.21.15.16,172.21.15.156,[v2:172.21.15.156:3301,v1:172.21.15.156:6790]" 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: fsid = f8999c8c-ab57-4aba-8d53-08e19e6ee3d5 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout:[osd] 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: osd journal size = 100 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub load threshold = 5.0 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: osd scrub max interval = 600 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: osd mclock profile = high_recovery_ops 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: osd recover clone overlap = true 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: osd recovery max chunk = 1048576 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug shutdown = true 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug op order = true 2024-04-24T23:47:55.721 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug verify stray on activate = true 2024-04-24T23:47:55.722 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug trim objects = true 2024-04-24T23:47:55.722 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.722 INFO:teuthology.orchestra.run.smithi156.stdout: osd open classes on start = true 2024-04-24T23:47:55.722 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug pg log writeout = true 2024-04-24T23:47:55.722 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.722 INFO:teuthology.orchestra.run.smithi156.stdout: osd deep scrub update digest min age = 30 2024-04-24T23:47:55.722 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.722 INFO:teuthology.orchestra.run.smithi156.stdout: osd map max advance = 10 2024-04-24T23:47:55.722 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.722 INFO:teuthology.orchestra.run.smithi156.stdout: journal zero on create = true 2024-04-24T23:47:55.722 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.722 INFO:teuthology.orchestra.run.smithi156.stdout: filestore ondisk finisher threads = 3 2024-04-24T23:47:55.722 INFO:teuthology.orchestra.run.smithi156.stdout: filestore apply finisher threads = 3 2024-04-24T23:47:55.722 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: bdev debug aio = true 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: osd debug misdirected ops = true 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore block size = 96636764160 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: bluestore fsck on mount = True 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: debug bluefs = 20 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: debug bluestore = 20 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: debug osd = 20 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: debug rocksdb = 10 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd backfillfull_ratio = 0.85 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd full ratio = 0.9 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd nearfull ratio = 0.8 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: osd failsafe full ratio = 0.95 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: osd objectstore = bluestore 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: osd op complaint time = 180 2024-04-24T23:47:55.723 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr] 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout: debug mgr = 20 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout: mgr/telemetry/nag = false 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout: debug client = 20 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout:[mon] 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout: debug mon = 20 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout: debug paxos = 20 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout: debug auth = 20 2024-04-24T23:47:55.724 INFO:teuthology.orchestra.run.smithi156.stdout: mon data avail warn = 5 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: mon mgr mkfs grace = 240 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min pgs per osd = 4 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd reporter subtree level = osd 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: mon osd prime pg temp = true 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: mon reweight min bytes per osd = 10 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: auth mon ticket ttl = 660 # 11m 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: auth service ticket ttl = 240 # 4m 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: # don't complain about insecure global_id in the test suite 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.725 INFO:teuthology.orchestra.run.smithi156.stdout: # 1m isn't quite enough 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout: mon_down_mkfs_grace = 2m 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout: mon_warn_on_filestore_osds = false 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout: mon down mkfs grace = 300 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout: mon op complaint time = 120 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout:[client] 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout: rgw cache enabled = true 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable ops log = true 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout: rgw enable usage log = true 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout: client mount timeout = 600 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout: debug client = 20 2024-04-24T23:47:55.726 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout: pid file = "" 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout: rados mon op timeout = 900 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout: rados osd op timeout = 900 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.a] 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.b] 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout:[mon.c] 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.a] 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.c] 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.b] 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.d] 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout:[mds] 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout: debug mds = 20 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout: debug mds balancer = 20 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout: debug ms = 1 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout: mds debug frag = True 2024-04-24T23:47:55.727 INFO:teuthology.orchestra.run.smithi156.stdout: mds debug scatterstat = True 2024-04-24T23:47:55.728 INFO:teuthology.orchestra.run.smithi156.stdout: mds op complaint time = 180 2024-04-24T23:47:55.728 INFO:teuthology.orchestra.run.smithi156.stdout: mds verify scatter = True 2024-04-24T23:47:55.728 INFO:teuthology.orchestra.run.smithi156.stdout: osd op complaint time = 180 2024-04-24T23:47:55.728 INFO:teuthology.orchestra.run.smithi156.stdout: rados mon op timeout = 900 2024-04-24T23:47:55.728 INFO:teuthology.orchestra.run.smithi156.stdout: rados osd op timeout = 900 2024-04-24T23:47:55.729 INFO:tasks.ceph:Creating admin key on mon.a... 2024-04-24T23:47:55.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-04-24T23:47:55.823 INFO:tasks.ceph:Copying monmap to all nodes... 2024-04-24T23:47:55.823 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:47:55.823 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-04-24T23:47:55.839 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:47:55.839 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-04-24T23:47:55.893 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:47:55.893 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:47:55.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T23:47:55.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T23:47:55.967 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:47:55.967 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:47:56.022 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:47:56.022 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:47:56.022 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.keyring 2024-04-24T23:47:56.023 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-04-24T23:47:56.056 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:47:56.056 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:47:56.113 INFO:tasks.ceph:Setting up mon nodes... 2024-04-24T23:47:56.113 INFO:tasks.ceph:Setting up mgr nodes... 2024-04-24T23:47:56.113 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-04-24T23:47:56.163 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-04-24T23:47:56.166 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-04-24T23:47:56.218 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-04-24T23:47:56.220 INFO:tasks.ceph:Setting up mds nodes... 2024-04-24T23:47:56.220 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-04-24T23:47:56.270 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-04-24T23:47:56.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-04-24T23:47:56.297 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-04-24T23:47:56.388 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-04-24T23:47:56.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-04-24T23:47:56.414 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-04-24T23:47:56.462 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-04-24T23:47:56.465 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-04-24T23:47:56.488 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-04-24T23:47:56.577 INFO:teuthology.orchestra.run.smithi156.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-04-24T23:47:56.579 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-04-24T23:47:56.602 INFO:tasks.ceph_client:Setting up client nodes... 2024-04-24T23:47:56.603 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-24T23:47:56.643 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-04-24T23:47:56.655 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-04-24T23:47:56.735 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-04-24T23:47:56.748 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-24T23:47:56.787 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-04-24T23:47:56.799 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-04-24T23:47:56.877 INFO:teuthology.orchestra.run.smithi156.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-04-24T23:47:56.889 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-04-24T23:47:56.890 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi156.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-04-24T23:47:56.890 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-04-24T23:47:56.914 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T23:47:56.914 INFO:tasks.ceph:role: osd.0 2024-04-24T23:47:56.915 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:47:56.915 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T23:47:56.978 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:47:56.978 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:47:56.978 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:47:56.978 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:47:56.979 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:47:56.979 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:47:56.979 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:47:56.979 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:47:56.979 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:47:56.979 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:47:57.010 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-04-24T23:47:57.012 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-04-24T23:47:57.012 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-04-24T23:47:57.081 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-04-24T23:47:57.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-04-24T23:47:57.213 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T23:47:57.213 INFO:tasks.ceph:role: osd.1 2024-04-24T23:47:57.214 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:47:57.214 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T23:47:57.277 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:47:57.278 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:47:57.278 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:47:57.278 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:47:57.278 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:47:57.278 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:47:57.278 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:47:57.278 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:47:57.278 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:47:57.278 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:47:57.315 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-04-24T23:47:57.317 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-04-24T23:47:57.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-04-24T23:47:57.386 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-04-24T23:47:57.454 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-04-24T23:47:57.519 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T23:47:57.519 INFO:tasks.ceph:role: osd.2 2024-04-24T23:47:57.519 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:47:57.519 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T23:47:57.583 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:47:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:47:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:47:57.583 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:47:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:47:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:47:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:47:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:47:57.584 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:47:57.584 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:47:57.618 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-04-24T23:47:57.620 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-04-24T23:47:57.620 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-04-24T23:47:57.688 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-04-24T23:47:57.757 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-04-24T23:47:57.821 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-04-24T23:47:57.821 INFO:tasks.ceph:role: osd.3 2024-04-24T23:47:57.821 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:47:57.822 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-24T23:47:57.885 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:47:57.885 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:47:57.885 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:47:57.885 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:47:57.885 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:47:57.885 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:47:57.885 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:47:57.885 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:47:57.885 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:47:57.885 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:47:57.920 INFO:teuthology.orchestra.run.smithi016.stdout:Discarding blocks...Done. 2024-04-24T23:47:57.922 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-04-24T23:47:57.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-04-24T23:47:57.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-04-24T23:47:58.059 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:47:58.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:47:58.153+0000 7fd17ea10640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-04-24T23:47:58.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:47:58.153+0000 7fd17ea10640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-04-24T23:47:58.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:47:58.153+0000 7fd17ea10640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T23:47:58.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:47:58.153+0000 7fd17ea10640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T23:47:58.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:47:58.153+0000 7fd17ea10640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-04-24T23:47:58.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:47:58.153+0000 7fd17ea10640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-04-24T23:48:01.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-24T23:48:01.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:48:01.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:01.126+0000 7f73b6820640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-04-24T23:48:01.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:01.126+0000 7f73b6820640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-04-24T23:48:01.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:01.126+0000 7f73b6820640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T23:48:01.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:01.126+0000 7f73b6820640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T23:48:01.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:01.126+0000 7f73b6820640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-04-24T23:48:01.127 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:01.127+0000 7f73b6820640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-04-24T23:48:03.976 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-24T23:48:04.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:48:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:04.095+0000 7fd710016640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-04-24T23:48:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:04.095+0000 7fd710016640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-04-24T23:48:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:04.095+0000 7fd710016640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T23:48:04.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:04.095+0000 7fd710016640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T23:48:04.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:04.095+0000 7fd710016640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-04-24T23:48:04.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:04.096+0000 7fd710016640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-04-24T23:48:06.941 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-24T23:48:06.970 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:48:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:07.061+0000 7f92a84ea640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-04-24T23:48:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:07.062+0000 7f92a84ea640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-04-24T23:48:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:07.062+0000 7f92a84ea640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T23:48:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:07.062+0000 7f92a84ea640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T23:48:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:07.062+0000 7f92a84ea640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-04-24T23:48:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:07.062+0000 7f92a84ea640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-04-24T23:48:09.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-24T23:48:09.941 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-04-24T23:48:09.970 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T23:48:09.970 INFO:tasks.ceph:role: osd.4 2024-04-24T23:48:09.970 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:48:09.970 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-04-24T23:48:10.038 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:48:10.038 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:48:10.038 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:48:10.038 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:48:10.038 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:48:10.038 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:48:10.038 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:48:10.038 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:48:10.038 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:48:10.039 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:48:10.070 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-04-24T23:48:10.072 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-04-24T23:48:10.072 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-04-24T23:48:10.152 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-04-24T23:48:10.223 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-04-24T23:48:10.288 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T23:48:10.288 INFO:tasks.ceph:role: osd.5 2024-04-24T23:48:10.288 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:48:10.289 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-04-24T23:48:10.352 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:48:10.352 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:48:10.352 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:48:10.352 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:48:10.353 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:48:10.353 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:48:10.353 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:48:10.353 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:48:10.353 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:48:10.353 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:48:10.385 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-04-24T23:48:10.387 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-04-24T23:48:10.387 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-04-24T23:48:10.456 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-04-24T23:48:10.525 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-04-24T23:48:10.590 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T23:48:10.590 INFO:tasks.ceph:role: osd.6 2024-04-24T23:48:10.590 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:48:10.590 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-04-24T23:48:10.669 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:48:10.670 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:48:10.670 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:48:10.670 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:48:10.670 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:48:10.670 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:48:10.670 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:48:10.670 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:48:10.670 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:48:10.670 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:48:10.703 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-04-24T23:48:10.705 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-04-24T23:48:10.705 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-04-24T23:48:10.773 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-04-24T23:48:10.841 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-04-24T23:48:10.905 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-04-24T23:48:10.905 INFO:tasks.ceph:role: osd.7 2024-04-24T23:48:10.905 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:48:10.905 DEBUG:teuthology.orchestra.run.smithi156:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-04-24T23:48:10.969 INFO:teuthology.orchestra.run.smithi156.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-04-24T23:48:10.969 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 attr=2, projid32bit=1 2024-04-24T23:48:10.969 INFO:teuthology.orchestra.run.smithi156.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-04-24T23:48:10.969 INFO:teuthology.orchestra.run.smithi156.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-04-24T23:48:10.969 INFO:teuthology.orchestra.run.smithi156.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-04-24T23:48:10.969 INFO:teuthology.orchestra.run.smithi156.stdout: = sunit=0 swidth=0 blks 2024-04-24T23:48:10.969 INFO:teuthology.orchestra.run.smithi156.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-04-24T23:48:10.970 INFO:teuthology.orchestra.run.smithi156.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-04-24T23:48:10.970 INFO:teuthology.orchestra.run.smithi156.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-04-24T23:48:10.970 INFO:teuthology.orchestra.run.smithi156.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-04-24T23:48:11.002 INFO:teuthology.orchestra.run.smithi156.stdout:Discarding blocks...Done. 2024-04-24T23:48:11.004 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi156.front.sepia.ceph.com -o noatime 2024-04-24T23:48:11.004 DEBUG:teuthology.orchestra.run.smithi156:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-04-24T23:48:11.071 DEBUG:teuthology.orchestra.run.smithi156:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-04-24T23:48:11.138 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:48:11.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:11.232+0000 7fb061441640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-04-24T23:48:11.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:11.232+0000 7fb061441640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-04-24T23:48:11.232 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:11.232+0000 7fb061441640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T23:48:11.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:11.232+0000 7fb061441640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T23:48:11.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:11.232+0000 7fb061441640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-04-24T23:48:11.233 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:11.233+0000 7fb061441640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-04-24T23:48:14.076 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-24T23:48:14.103 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:48:14.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:14.194+0000 7f3365616640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-04-24T23:48:14.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:14.194+0000 7f3365616640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-04-24T23:48:14.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:14.194+0000 7f3365616640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T23:48:14.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:14.194+0000 7f3365616640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T23:48:14.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:14.194+0000 7f3365616640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-04-24T23:48:14.194 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:14.194+0000 7f3365616640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-04-24T23:48:17.044 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-24T23:48:17.073 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:48:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:17.161+0000 7f6ecf540640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-04-24T23:48:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:17.161+0000 7f6ecf540640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-04-24T23:48:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:17.161+0000 7f6ecf540640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-24T23:48:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:17.161+0000 7f6ecf540640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-24T23:48:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:17.161+0000 7f6ecf540640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-04-24T23:48:17.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:17.161+0000 7f6ecf540640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-04-24T23:48:20.021 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-24T23:48:20.049 DEBUG:teuthology.orchestra.run.smithi156:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:48:20.137 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:20.138+0000 7fa9c2a86640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-04-24T23:48:20.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:20.138+0000 7fa9c2a86640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-04-24T23:48:20.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:20.138+0000 7fa9c2a86640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-24T23:48:20.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:20.138+0000 7fa9c2a86640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-24T23:48:20.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:20.138+0000 7fa9c2a86640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-04-24T23:48:20.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:20.138+0000 7fa9c2a86640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-04-24T23:48:22.979 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-24T23:48:23.008 INFO:tasks.ceph:Reading keys from all nodes... 2024-04-24T23:48:23.008 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:48:23.008 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-04-24T23:48:23.038 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:48:23.038 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-04-24T23:48:23.106 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:48:23.106 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-04-24T23:48:23.174 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:48:23.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-04-24T23:48:23.242 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:48:23.242 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-04-24T23:48:23.310 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:48:23.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-04-24T23:48:23.376 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:48:23.376 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-04-24T23:48:23.442 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:48:23.443 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-04-24T23:48:23.471 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:48:23.472 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-04-24T23:48:23.539 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:48:23.539 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-04-24T23:48:23.605 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:48:23.606 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-04-24T23:48:23.671 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:48:23.671 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-04-24T23:48:23.736 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:48:23.736 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-04-24T23:48:23.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:48:23.789 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:48:23.801 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:48:23.801 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-04-24T23:48:23.904 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:48:23.904 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-04-24T23:48:23.921 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:48:23.921 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-04-24T23:48:23.978 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:48:23.978 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-04-24T23:48:23.994 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:48:23.994 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-04-24T23:48:24.050 INFO:tasks.ceph:Adding keys to all mons... 2024-04-24T23:48:24.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-24T23:48:24.054 DEBUG:teuthology.orchestra.run.smithi156:> sudo tee -a /etc/ceph/ceph.keyring 2024-04-24T23:48:24.094 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.y] 2024-04-24T23:48:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAsmilmrpjUCRAAfH4AjI/pUjQg1/mn7GpDTw== 2024-04-24T23:48:24.094 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a] 2024-04-24T23:48:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAsmilmGXU4EBAAot69Ad+4q0ajnN525mELrA== 2024-04-24T23:48:24.094 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.c] 2024-04-24T23:48:24.094 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAsmilmhlw3FxAAi8h6MHJf5CvKyTOCBTYbkA== 2024-04-24T23:48:24.094 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.0] 2024-04-24T23:48:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAumilmBjEvCRAAGfXfrApvnEA4UsH2LLrOtw== 2024-04-24T23:48:24.095 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.1] 2024-04-24T23:48:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAxmilmyHSaBxAAo8nFyeMCWXSoZsQ0BpeOnQ== 2024-04-24T23:48:24.095 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.2] 2024-04-24T23:48:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQA0milmi8e/BRAAKHTiaAzRVBDzUbMETp2Z3Q== 2024-04-24T23:48:24.095 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.3] 2024-04-24T23:48:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQA3milmnu29AxAA1obUjImx0vtxWob+Rg9tVA== 2024-04-24T23:48:24.095 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.x] 2024-04-24T23:48:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAsmilmKBoWDRAAYsVUTv84qwU3VeIhhLNCTw== 2024-04-24T23:48:24.095 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b] 2024-04-24T23:48:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAsmilmwriuGxAA8IeWbl5CQnyegSGw47x7Qg== 2024-04-24T23:48:24.095 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.d] 2024-04-24T23:48:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAsmilm+Dt+IhAAPk+MsjISFV980KLFyT5LiA== 2024-04-24T23:48:24.096 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.4] 2024-04-24T23:48:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQA7milmFHvpDRAAOIPyH67azej6WG7kLQR/Dw== 2024-04-24T23:48:24.096 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.5] 2024-04-24T23:48:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQA+milmEWmcCxAAPY/2viwca5G+6T6JWIjkDQ== 2024-04-24T23:48:24.096 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.6] 2024-04-24T23:48:24.096 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBBmilmtjSoCRAAjtwouhdVLqMb6OrTuq1nUw== 2024-04-24T23:48:24.096 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.7] 2024-04-24T23:48:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBEmilmDB1ICBAAH5A6jnsXpdpw46yRLIigCg== 2024-04-24T23:48:24.097 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-04-24T23:48:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAsmilm81VpJhAAXQnTDv4ShV2nH4Tpv5oziw== 2024-04-24T23:48:24.097 INFO:teuthology.orchestra.run.smithi016.stdout:[client.2] 2024-04-24T23:48:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAsmilmC+jxKxAAt1KAO9bdg0WndKIqvNJBCA== 2024-04-24T23:48:24.097 INFO:teuthology.orchestra.run.smithi016.stdout:[client.1] 2024-04-24T23:48:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAsmilmfVgCLxAAqmZ7wgN04zvzOcdv/Q6CVQ== 2024-04-24T23:48:24.097 INFO:teuthology.orchestra.run.smithi016.stdout:[client.3] 2024-04-24T23:48:24.097 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAsmilmpkNrNBAAfW/Xales/f5k4EgzHcqnkQ== 2024-04-24T23:48:24.113 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr.y] 2024-04-24T23:48:24.113 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAsmilmrpjUCRAAfH4AjI/pUjQg1/mn7GpDTw== 2024-04-24T23:48:24.113 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.a] 2024-04-24T23:48:24.113 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAsmilmGXU4EBAAot69Ad+4q0ajnN525mELrA== 2024-04-24T23:48:24.113 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.c] 2024-04-24T23:48:24.114 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAsmilmhlw3FxAAi8h6MHJf5CvKyTOCBTYbkA== 2024-04-24T23:48:24.114 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.0] 2024-04-24T23:48:24.114 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAumilmBjEvCRAAGfXfrApvnEA4UsH2LLrOtw== 2024-04-24T23:48:24.114 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.1] 2024-04-24T23:48:24.114 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAxmilmyHSaBxAAo8nFyeMCWXSoZsQ0BpeOnQ== 2024-04-24T23:48:24.114 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.2] 2024-04-24T23:48:24.114 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQA0milmi8e/BRAAKHTiaAzRVBDzUbMETp2Z3Q== 2024-04-24T23:48:24.114 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.3] 2024-04-24T23:48:24.114 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQA3milmnu29AxAA1obUjImx0vtxWob+Rg9tVA== 2024-04-24T23:48:24.114 INFO:teuthology.orchestra.run.smithi156.stdout:[mgr.x] 2024-04-24T23:48:24.114 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAsmilmKBoWDRAAYsVUTv84qwU3VeIhhLNCTw== 2024-04-24T23:48:24.114 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.b] 2024-04-24T23:48:24.114 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAsmilmwriuGxAA8IeWbl5CQnyegSGw47x7Qg== 2024-04-24T23:48:24.114 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.d] 2024-04-24T23:48:24.115 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAsmilm+Dt+IhAAPk+MsjISFV980KLFyT5LiA== 2024-04-24T23:48:24.115 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.4] 2024-04-24T23:48:24.115 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQA7milmFHvpDRAAOIPyH67azej6WG7kLQR/Dw== 2024-04-24T23:48:24.115 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.5] 2024-04-24T23:48:24.115 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQA+milmEWmcCxAAPY/2viwca5G+6T6JWIjkDQ== 2024-04-24T23:48:24.115 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.6] 2024-04-24T23:48:24.115 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBBmilmtjSoCRAAjtwouhdVLqMb6OrTuq1nUw== 2024-04-24T23:48:24.115 INFO:teuthology.orchestra.run.smithi156.stdout:[osd.7] 2024-04-24T23:48:24.115 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQBEmilmDB1ICBAAH5A6jnsXpdpw46yRLIigCg== 2024-04-24T23:48:24.115 INFO:teuthology.orchestra.run.smithi156.stdout:[client.0] 2024-04-24T23:48:24.115 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAsmilm81VpJhAAXQnTDv4ShV2nH4Tpv5oziw== 2024-04-24T23:48:24.115 INFO:teuthology.orchestra.run.smithi156.stdout:[client.2] 2024-04-24T23:48:24.115 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAsmilmC+jxKxAAt1KAO9bdg0WndKIqvNJBCA== 2024-04-24T23:48:24.115 INFO:teuthology.orchestra.run.smithi156.stdout:[client.1] 2024-04-24T23:48:24.116 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAsmilmfVgCLxAAqmZ7wgN04zvzOcdv/Q6CVQ== 2024-04-24T23:48:24.116 INFO:teuthology.orchestra.run.smithi156.stdout:[client.3] 2024-04-24T23:48:24.116 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQAsmilmpkNrNBAAfW/Xales/f5k4EgzHcqnkQ== 2024-04-24T23:48:24.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T23:48:24.141 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T23:48:24.207 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:48:24.209 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:48:24.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:48:24.263 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:48:24.314 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.317 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.365 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.367 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.415 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.418 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.470 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.520 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T23:48:24.523 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-04-24T23:48:24.571 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:48:24.574 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:48:24.625 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:48:24.628 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-04-24T23:48:24.680 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.682 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.732 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.734 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.782 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.784 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.833 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.835 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-04-24T23:48:24.884 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T23:48:24.886 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T23:48:24.937 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T23:48:24.939 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T23:48:24.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T23:48:24.992 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T23:48:25.044 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T23:48:25.046 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-04-24T23:48:25.098 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-04-24T23:48:25.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-04-24T23:48:25.123 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T23:48:25.218 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-04-24T23:48:25.242 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-04-24T23:48:25.266 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T23:48:25.361 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-04-24T23:48:25.385 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-04-24T23:48:25.449 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-04-24T23:48:25.542 DEBUG:teuthology.orchestra.run.smithi156:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-04-24T23:48:25.565 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:48:25.568 DEBUG:teuthology.orchestra.run.smithi156:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-04-24T23:48:25.621 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-04-24T23:48:25.621 INFO:tasks.ceph.mon.a:Restarting daemon 2024-04-24T23:48:25.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-04-24T23:48:25.624 INFO:tasks.ceph.mon.a:Started 2024-04-24T23:48:25.624 INFO:tasks.ceph.mon.b:Restarting daemon 2024-04-24T23:48:25.624 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-04-24T23:48:25.663 INFO:tasks.ceph.mon.b:Started 2024-04-24T23:48:25.663 INFO:tasks.ceph.mon.c:Restarting daemon 2024-04-24T23:48:25.664 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-04-24T23:48:25.665 INFO:tasks.ceph.mon.c:Started 2024-04-24T23:48:25.666 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-04-24T23:48:25.666 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-04-24T23:48:25.666 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-04-24T23:48:25.668 INFO:tasks.ceph.mgr.y:Started 2024-04-24T23:48:25.668 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-04-24T23:48:25.668 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-04-24T23:48:25.670 INFO:tasks.ceph.mgr.x:Started 2024-04-24T23:48:25.670 INFO:tasks.ceph:Setting crush tunables to default 2024-04-24T23:48:25.671 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd crush tunables default 2024-04-24T23:48:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.772+0000 7f07ab2cd640 1 Processor -- start 2024-04-24T23:48:25.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.772+0000 7f07ab2cd640 1 -- start start 2024-04-24T23:48:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07ab2cd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 0x7f07a40d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07ab2cd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 0x7f07a40cec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07ab2cd640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07a40b58e0 con 0x7f07a4162740 2024-04-24T23:48:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07ab2cd640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f07a40d09b0 con 0x7f07a40b5500 2024-04-24T23:48:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07ab2cd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f07a415ef70 con 0x7f07a40ae4c0 2024-04-24T23:48:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07a8868640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f07a4162740 0x7f07a40addb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39828/0 (socket says 172.21.15.16:39828) 2024-04-24T23:48:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07a8868640 1 -- 172.21.15.16:0/1678303837 learned_addr learned my addr 172.21.15.16:0/1678303837 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:48:25.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 0x7f07a40d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 0x7f07a40cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 msgr2=0x7f07a40d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-24T23:48:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 msgr2=0x7f07a40d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-24T23:48:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 0x7f07a40d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:48:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07a986a640 1 -- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 msgr2=0x7f07a40cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-24T23:48:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07a986a640 1 -- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 msgr2=0x7f07a40cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-24T23:48:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 0x7f07a40cec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:48:25.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 0x7f07a40d0470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T23:48:25.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.773+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 0x7f07a40cec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T23:48:25.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.974+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 0x7f07a40cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:25.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.974+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 0x7f07a40d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:25.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.974+0000 7f07a986a640 1 -- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 msgr2=0x7f07a40cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-24T23:48:25.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.974+0000 7f07a986a640 1 -- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 msgr2=0x7f07a40cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-24T23:48:25.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.974+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 0x7f07a40cec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:48:25.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.974+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 0x7f07a40cec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T23:48:25.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.974+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 msgr2=0x7f07a40d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-24T23:48:25.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.974+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 msgr2=0x7f07a40d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-24T23:48:25.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.974+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 0x7f07a40d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:48:25.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:25.974+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 0x7f07a40d0470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T23:48:26.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:26.375+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 0x7f07a40cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:26.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:26.375+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 0x7f07a40d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:26.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:26.375+0000 7f07a986a640 1 -- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 msgr2=0x7f07a40cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-24T23:48:26.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:26.375+0000 7f07a986a640 1 -- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 msgr2=0x7f07a40cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-24T23:48:26.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:26.375+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 0x7f07a40cec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:48:26.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:26.375+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 msgr2=0x7f07a40d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-24T23:48:26.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:26.375+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 msgr2=0x7f07a40d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-24T23:48:26.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:26.375+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 0x7f07a40cec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-24T23:48:26.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:26.375+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 0x7f07a40d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:48:26.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:26.375+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 0x7f07a40d0470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-24T23:48:27.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:27.176+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 0x7f07a40d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:27.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:27.176+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 0x7f07a40cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:27.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:27.176+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 msgr2=0x7f07a40d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-24T23:48:27.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:27.176+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 msgr2=0x7f07a40d0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-24T23:48:27.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:27.176+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 0x7f07a40d0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:48:27.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:27.176+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 0x7f07a40d0470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-24T23:48:27.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:27.176+0000 7f07a986a640 1 -- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 msgr2=0x7f07a40cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-04-24T23:48:27.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:27.176+0000 7f07a986a640 1 -- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 msgr2=0x7f07a40cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-24T23:48:27.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:27.176+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 0x7f07a40cec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:48:27.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:27.176+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 0x7f07a40cec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-24T23:48:28.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.773+0000 7f07a2ffd640 1 -- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 msgr2=0x7f07a40cec90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:48:28.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.773+0000 7f07a2ffd640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f07a40ae4c0 0x7f07a40cec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:28.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.773+0000 7f07a2ffd640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 msgr2=0x7f07a40d0470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:48:28.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.773+0000 7f07a2ffd640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.156:3300/0 conn(0x7f07a40b5500 0x7f07a40d0470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:28.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.773+0000 7f07a2ffd640 1 -- 172.21.15.16:0/1678303837 >> v1:172.21.15.16:6789/0 conn(0x7f07a4162740 legacy=0x7f07a40addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:28.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.773+0000 7f07a2ffd640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 0x7f0790005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:28.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.773+0000 7f07a2ffd640 1 -- 172.21.15.16:0/1678303837 --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07a415f250 con 0x7f0790006170 2024-04-24T23:48:28.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.773+0000 7f07a2ffd640 1 -- 172.21.15.16:0/1678303837 --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f07a40d09b0 con 0x7f07900036e0 2024-04-24T23:48:28.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.773+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 0x7f0790005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:28.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.773+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 msgr2=0x7f0790005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-24T23:48:28.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.773+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 msgr2=0x7f0790005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-24T23:48:28.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.773+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 0x7f0790005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:48:28.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.773+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 0x7f0790005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T23:48:28.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.974+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 0x7f0790005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:28.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.974+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 msgr2=0x7f0790005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-24T23:48:28.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.974+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 msgr2=0x7f0790005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-24T23:48:28.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.974+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 0x7f0790005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:48:28.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:28.974+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 0x7f0790005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T23:48:29.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:29.375+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 0x7f0790005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:29.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:29.375+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 msgr2=0x7f0790005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-24T23:48:29.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:29.375+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 msgr2=0x7f0790005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-24T23:48:29.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:29.375+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 0x7f0790005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:48:29.375 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:29.375+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 0x7f0790005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-24T23:48:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.177+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 0x7f0790005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.177+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 msgr2=0x7f0790005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-04-24T23:48:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.177+0000 7f07a9069640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 msgr2=0x7f0790005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-04-24T23:48:30.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.177+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 0x7f0790005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T23:48:30.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.177+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 0x7f0790005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-24T23:48:30.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.935+0000 7f07a3fff640 1 -- 172.21.15.16:0/1678303837 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2930001667 0 0) 0x7f07a415f250 con 0x7f0790006170 2024-04-24T23:48:30.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.935+0000 7f07a3fff640 1 -- 172.21.15.16:0/1678303837 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f078c003520 con 0x7f0790006170 2024-04-24T23:48:30.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a3fff640 1 -- 172.21.15.16:0/1678303837 <== mon.1 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 248616046 0 0) 0x7f078c003520 con 0x7f0790006170 2024-04-24T23:48:30.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a3fff640 1 -- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 msgr2=0x7f0790005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:48:30.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a3fff640 1 --2- 172.21.15.16:0/1678303837 >> v2:172.21.15.16:3300/0 conn(0x7f07900036e0 0x7f0790005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:30.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a3fff640 1 -- 172.21.15.16:0/1678303837 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079000af50 con 0x7f0790006170 2024-04-24T23:48:30.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a3fff640 1 -- 172.21.15.16:0/1678303837 <== mon.1 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f0798001710 con 0x7f0790006170 2024-04-24T23:48:30.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a3fff640 1 -- 172.21.15.16:0/1678303837 >> v1:172.21.15.156:6789/0 conn(0x7f0790006170 legacy=0x7f0790008610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:30.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a986a640 1 -- 172.21.15.16:0/1678303837 reap_dead start 2024-04-24T23:48:30.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a3fff640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f078c003a00 0x7f078c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:30.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a3fff640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f078c004800 0x7f078c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:30.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a3fff640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f078c0053c0 0x7f078c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:30.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a3fff640 1 -- 172.21.15.16:0/1678303837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07a40d09b0 con 0x7f078c003a00 2024-04-24T23:48:30.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a3fff640 1 -- 172.21.15.16:0/1678303837 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f078c00c280 con 0x7f078c0053c0 2024-04-24T23:48:30.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a3fff640 1 -- 172.21.15.16:0/1678303837 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f078c00c480 con 0x7f078c004800 2024-04-24T23:48:30.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a9069640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f078c003a00 0x7f078c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:30.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f078c0053c0 0x7f078c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:30.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.936+0000 7f07a8868640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f078c004800 0x7f078c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:30.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.937+0000 7f07a986a640 1 -- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f078c004800 msgr2=0x7f078c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:30.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.937+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f078c004800 0x7f078c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:30.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.937+0000 7f07a986a640 1 -- 172.21.15.16:0/1678303837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f078c003a00 msgr2=0x7f078c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:30.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.937+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f078c003a00 0x7f078c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:30.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.937+0000 7f07a986a640 1 -- 172.21.15.16:0/1678303837 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078c003520 con 0x7f078c0053c0 2024-04-24T23:48:30.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.938+0000 7f07a986a640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f078c0053c0 0x7f078c00b9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f07a40b39d0 tx=0x7f078c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b514a1bb66bc79a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:30.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.938+0000 7f07a3fff640 1 -- 172.21.15.16:0/1678303837 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0794031070 con 0x7f078c0053c0 2024-04-24T23:48:30.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.938+0000 7f07a3fff640 1 -- 172.21.15.16:0/1678303837 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f079402b7b0 con 0x7f078c0053c0 2024-04-24T23:48:30.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.938+0000 7f07a3fff640 1 -- 172.21.15.16:0/1678303837 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0794003db0 con 0x7f078c0053c0 2024-04-24T23:48:30.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.939+0000 7f07ab2cd640 1 -- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f078c0053c0 msgr2=0x7f078c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:30.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.939+0000 7f07ab2cd640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f078c0053c0 0x7f078c00b9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f07a40b39d0 tx=0x7f078c005830 comp rx=0 tx=0).stop 2024-04-24T23:48:30.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.939+0000 7f07ab2cd640 1 -- 172.21.15.16:0/1678303837 shutdown_connections 2024-04-24T23:48:30.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.939+0000 7f07ab2cd640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f078c0053c0 0x7f078c00b9c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:30.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.939+0000 7f07ab2cd640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f078c004800 0x7f078c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:30.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.939+0000 7f07ab2cd640 1 --2- 172.21.15.16:0/1678303837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f078c003a00 0x7f078c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:30.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.939+0000 7f07ab2cd640 1 -- 172.21.15.16:0/1678303837 >> 172.21.15.16:0/1678303837 conn(0x7f07a40b2910 msgr2=0x7f07a40b2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:30.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.939+0000 7f07ab2cd640 1 -- 172.21.15.16:0/1678303837 shutdown_connections 2024-04-24T23:48:30.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.939+0000 7f07ab2cd640 1 -- 172.21.15.16:0/1678303837 wait complete. 2024-04-24T23:48:30.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.941+0000 7f07ab2cd640 1 Processor -- start 2024-04-24T23:48:30.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.941+0000 7f07ab2cd640 1 -- start start 2024-04-24T23:48:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07ab2cd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f078c003a00 0x7f07a41f39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07ab2cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f078c004800 0x7f07a41f79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07ab2cd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f078c0053c0 0x7f07a4154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07ab2cd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f07a4162220 con 0x7f078c004800 2024-04-24T23:48:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07ab2cd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f07a4161f40 con 0x7f078c003a00 2024-04-24T23:48:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07ab2cd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f07a40aea30 con 0x7f078c0053c0 2024-04-24T23:48:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07a8868640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f078c004800 0x7f07a41f79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07a8868640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f078c004800 0x7f07a41f79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59768/0 (socket says 172.21.15.16:59768) 2024-04-24T23:48:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07a8868640 1 -- 172.21.15.16:0/495487965 learned_addr learned my addr 172.21.15.16:0/495487965 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:30.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07a9069640 1 --2- 172.21.15.16:0/495487965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f078c003a00 0x7f07a41f39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07a986a640 1 --2- 172.21.15.16:0/495487965 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f078c0053c0 0x7f07a4154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07a8868640 1 -- 172.21.15.16:0/495487965 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f078c0053c0 msgr2=0x7f07a4154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07a8868640 1 --2- 172.21.15.16:0/495487965 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f078c0053c0 0x7f07a4154010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07a8868640 1 -- 172.21.15.16:0/495487965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f078c003a00 msgr2=0x7f07a41f39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07a8868640 1 --2- 172.21.15.16:0/495487965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f078c003a00 0x7f07a41f39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.942+0000 7f07a8868640 1 -- 172.21.15.16:0/495487965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a406b970 con 0x7f078c004800 2024-04-24T23:48:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.943+0000 7f07a8868640 1 --2- 172.21.15.16:0/495487965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f078c004800 0x7f07a41f79c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0798007290 tx=0x7f0798007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.943+0000 7f07a1ffb640 1 -- 172.21.15.16:0/495487965 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079800f940 con 0x7f078c004800 2024-04-24T23:48:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.943+0000 7f07a1ffb640 1 -- 172.21.15.16:0/495487965 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0798018070 con 0x7f078c004800 2024-04-24T23:48:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.943+0000 7f07a1ffb640 1 -- 172.21.15.16:0/495487965 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079800fd40 con 0x7f078c004800 2024-04-24T23:48:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.943+0000 7f07ab2cd640 1 -- 172.21.15.16:0/495487965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07a406bc10 con 0x7f078c004800 2024-04-24T23:48:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.943+0000 7f07a1ffb640 1 -- 172.21.15.16:0/495487965 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f079801d460 con 0x7f078c004800 2024-04-24T23:48:30.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.943+0000 7f07ab2cd640 1 -- 172.21.15.16:0/495487965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a406c1d0 con 0x7f078c004800 2024-04-24T23:48:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.944+0000 7f07a1ffb640 1 -- 172.21.15.16:0/495487965 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f079801d930 con 0x7f078c004800 2024-04-24T23:48:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.945+0000 7f07ab2cd640 1 -- 172.21.15.16:0/495487965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07a4068f90 con 0x7f078c004800 2024-04-24T23:48:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:30.947+0000 7f07a1ffb640 1 -- 172.21.15.16:0/495487965 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0798013760 con 0x7f078c004800 2024-04-24T23:48:31.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:31.016+0000 7f07ab2cd640 1 -- 172.21.15.16:0/495487965 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f07a40b5f60 con 0x7f078c004800 2024-04-24T23:48:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:31.017+0000 7f07a1ffb640 1 -- 172.21.15.16:0/495487965 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f079801db30 con 0x7f078c004800 2024-04-24T23:48:31.017 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-04-24T23:48:31.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:31.018+0000 7f07ab2cd640 1 -- 172.21.15.16:0/495487965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f078c004800 msgr2=0x7f07a41f79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:31.018+0000 7f07ab2cd640 1 --2- 172.21.15.16:0/495487965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f078c004800 0x7f07a41f79c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0798007290 tx=0x7f0798007750 comp rx=0 tx=0).stop 2024-04-24T23:48:31.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:31.018+0000 7f07ab2cd640 1 -- 172.21.15.16:0/495487965 shutdown_connections 2024-04-24T23:48:31.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:31.018+0000 7f07ab2cd640 1 --2- 172.21.15.16:0/495487965 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f078c0053c0 0x7f07a4154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:31.018+0000 7f07ab2cd640 1 --2- 172.21.15.16:0/495487965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f078c004800 0x7f07a41f79c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:31.018+0000 7f07ab2cd640 1 --2- 172.21.15.16:0/495487965 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f078c003a00 0x7f07a41f39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:31.018+0000 7f07ab2cd640 1 -- 172.21.15.16:0/495487965 >> 172.21.15.16:0/495487965 conn(0x7f07a40b2910 msgr2=0x7f07a40b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:31.018+0000 7f07ab2cd640 1 -- 172.21.15.16:0/495487965 shutdown_connections 2024-04-24T23:48:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:31.018+0000 7f07ab2cd640 1 -- 172.21.15.16:0/495487965 wait complete. 2024-04-24T23:48:31.030 INFO:tasks.ceph:check_enable_crimson: False 2024-04-24T23:48:31.030 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-04-24T23:48:31.030 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:48:31.030 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-04-24T23:48:31.055 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:48:31.055 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-04-24T23:48:31.057 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:31.057+0000 7fcfc9212040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T23:48:31.061 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:31.062+0000 7fac8b91c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T23:48:31.085 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:48:31.085 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-04-24T23:48:31.149 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:48:31.149 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-04-24T23:48:31.204 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:31.204+0000 7fcfc9212040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:48:31.214 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:48:31.214 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-04-24T23:48:31.217 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:31.216+0000 7fac8b91c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:48:31.237 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:48:31.237 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-04-24T23:48:31.260 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:31.260+0000 7fcfc9212040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T23:48:31.273 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:31.272+0000 7fac8b91c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T23:48:31.282 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:48:31.282 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-04-24T23:48:31.346 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:48:31.346 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-04-24T23:48:31.410 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 6f0cd63f-5346-49f1-a49c-909e5c367d59 0 2024-04-24T23:48:31.520 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:31.521+0000 7fcfc9212040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T23:48:31.526 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:31.526+0000 7fac8b91c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T23:48:31.551 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.549+0000 7f2902bdc640 1 Processor -- start 2024-04-24T23:48:31.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.549+0000 7f2902bdc640 1 -- start start 2024-04-24T23:48:31.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.549+0000 7f2902bdc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f28fc0b5500 0x7f28fc0d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:31.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.549+0000 7f2902bdc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28fc0ae4c0 0x7f28fc0cec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:31.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.549+0000 7f2902bdc640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28fc0b58e0 con 0x7f28fc162740 2024-04-24T23:48:31.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.549+0000 7f2902bdc640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f28fc0d09b0 con 0x7f28fc0b5500 2024-04-24T23:48:31.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.549+0000 7f2902bdc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f28fc15ef70 con 0x7f28fc0ae4c0 2024-04-24T23:48:31.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.550+0000 7f2900978640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f28fc0b5500 0x7f28fc0d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:31.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.550+0000 7f2901179640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28fc0ae4c0 0x7f28fc0cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:31.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.550+0000 7f2900978640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f28fc0b5500 0x7f28fc0d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:54944/0 (socket says 172.21.15.156:54944) 2024-04-24T23:48:31.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.550+0000 7f2900978640 1 -- 172.21.15.156:0/2078449009 learned_addr learned my addr 172.21.15.156:0/2078449009 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-24T23:48:31.552 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.550+0000 7f2901179640 1 --2- 172.21.15.156:0/2078449009 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28fc0ae4c0 0x7f28fc0cec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:46546/0 (socket says 172.21.15.156:46546) 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.550+0000 7f2900978640 1 -- 172.21.15.156:0/2078449009 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28fc0ae4c0 msgr2=0x7f28fc0cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.550+0000 7f2900978640 1 --2- 172.21.15.156:0/2078449009 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28fc0ae4c0 0x7f28fc0cec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.550+0000 7f2900978640 1 -- 172.21.15.156:0/2078449009 >> v1:172.21.15.156:6789/0 conn(0x7f28fc162740 legacy=0x7f28fc0addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.550+0000 7f2900978640 1 -- 172.21.15.156:0/2078449009 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28fc15f250 con 0x7f28fc0b5500 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.550+0000 7f2901179640 1 --2- 172.21.15.156:0/2078449009 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28fc0ae4c0 0x7f28fc0cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.551+0000 7f2900978640 1 --2- 172.21.15.156:0/2078449009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28fc0b5500 0x7f28fc0d0470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f28f0009870 tx=0x7f28f002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de3c5df9218e101 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.551+0000 7f28fb7fe640 1 -- 172.21.15.156:0/2078449009 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f0037070 con 0x7f28fc0b5500 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.551+0000 7f28fb7fe640 1 -- 172.21.15.156:0/2078449009 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28f0032440 con 0x7f28fc0b5500 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.551+0000 7f28fb7fe640 1 -- 172.21.15.156:0/2078449009 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f0032810 con 0x7f28fc0b5500 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.551+0000 7f2902bdc640 1 -- 172.21.15.156:0/2078449009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28fc0b5500 msgr2=0x7f28fc0d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.551+0000 7f2902bdc640 1 --2- 172.21.15.156:0/2078449009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28fc0b5500 0x7f28fc0d0470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f28f0009870 tx=0x7f28f002a270 comp rx=0 tx=0).stop 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.551+0000 7f2902bdc640 1 -- 172.21.15.156:0/2078449009 shutdown_connections 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.551+0000 7f2902bdc640 1 --2- 172.21.15.156:0/2078449009 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28fc0ae4c0 0x7f28fc0cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.551+0000 7f2902bdc640 1 --2- 172.21.15.156:0/2078449009 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28fc0b5500 0x7f28fc0d0470 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.553 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.551+0000 7f2902bdc640 1 -- 172.21.15.156:0/2078449009 >> 172.21.15.156:0/2078449009 conn(0x7f28fc0b2910 msgr2=0x7f28fc0b2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:31.554 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.552+0000 7f2902bdc640 1 -- 172.21.15.156:0/2078449009 shutdown_connections 2024-04-24T23:48:31.554 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.552+0000 7f2902bdc640 1 -- 172.21.15.156:0/2078449009 wait complete. 2024-04-24T23:48:31.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.553+0000 7f2902bdc640 1 Processor -- start 2024-04-24T23:48:31.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.553+0000 7f2902bdc640 1 -- start start 2024-04-24T23:48:31.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2902bdc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28fc0ae4c0 0x7f28fc0b15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:31.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2902bdc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28fc0b5500 0x7f28fc15c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:31.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2902bdc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28fc162740 0x7f28fc1503d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:31.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2902bdc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc162220 con 0x7f28fc0ae4c0 2024-04-24T23:48:31.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2902bdc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f28fc161f40 con 0x7f28fc0b5500 2024-04-24T23:48:31.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2902bdc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f28fc0aea30 con 0x7f28fc162740 2024-04-24T23:48:31.556 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2901179640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28fc162740 0x7f28fc1503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f28fbfff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28fc0b5500 0x7f28fc15c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2901179640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28fc162740 0x7f28fc1503d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:46550/0 (socket says 172.21.15.156:46550) 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2901179640 1 -- 172.21.15.156:0/1813714338 learned_addr learned my addr 172.21.15.156:0/1813714338 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2900978640 1 --2- 172.21.15.156:0/1813714338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28fc0ae4c0 0x7f28fc0b15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2901179640 1 -- 172.21.15.156:0/1813714338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28fc0b5500 msgr2=0x7f28fc15c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2901179640 1 --2- 172.21.15.156:0/1813714338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28fc0b5500 0x7f28fc15c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2901179640 1 -- 172.21.15.156:0/1813714338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28fc0ae4c0 msgr2=0x7f28fc0b15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2901179640 1 --2- 172.21.15.156:0/1813714338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28fc0ae4c0 0x7f28fc0b15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2901179640 1 -- 172.21.15.156:0/1813714338 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28fc1e7ae0 con 0x7f28fc162740 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2901179640 1 --2- 172.21.15.156:0/1813714338 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28fc162740 0x7f28fc1503d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28f400efc0 tx=0x7f28f4009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f28f97fa640 1 -- 172.21.15.156:0/1813714338 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4018030 con 0x7f28fc162740 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f28f97fa640 1 -- 172.21.15.156:0/1813714338 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28f400cde0 con 0x7f28fc162740 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f28f97fa640 1 -- 172.21.15.156:0/1813714338 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28f4003bb0 con 0x7f28fc162740 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2902bdc640 1 -- 172.21.15.156:0/1813714338 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28fc1e7dc0 con 0x7f28fc162740 2024-04-24T23:48:31.557 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.554+0000 7f2902bdc640 1 -- 172.21.15.156:0/1813714338 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f28fc1e8380 con 0x7f28fc162740 2024-04-24T23:48:31.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.555+0000 7f28f97fa640 1 -- 172.21.15.156:0/1813714338 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f28f401b070 con 0x7f28fc162740 2024-04-24T23:48:31.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.555+0000 7f28f97fa640 1 -- 172.21.15.156:0/1813714338 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f28f401d910 con 0x7f28fc162740 2024-04-24T23:48:31.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.555+0000 7f2902bdc640 1 -- 172.21.15.156:0/1813714338 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28fc068f90 con 0x7f28fc162740 2024-04-24T23:48:31.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.558+0000 7f28f97fa640 1 -- 172.21.15.156:0/1813714338 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f28f4010040 con 0x7f28fc162740 2024-04-24T23:48:31.569 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:31.570+0000 7fcfc9212040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T23:48:31.574 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:31.574+0000 7fac8b91c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T23:48:31.630 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.630+0000 7f2902bdc640 1 -- 172.21.15.156:0/1813714338 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6f0cd63f-5346-49f1-a49c-909e5c367d59", "id": 0} v 0) -- 0x7f28fc0b5f60 con 0x7f28fc162740 2024-04-24T23:48:31.633 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.633+0000 7f28f97fa640 1 -- 172.21.15.156:0/1813714338 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6f0cd63f-5346-49f1-a49c-909e5c367d59", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f28f401dbe0 con 0x7f28fc162740 2024-04-24T23:48:31.633 INFO:teuthology.orchestra.run.smithi156.stdout:0 2024-04-24T23:48:31.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.634+0000 7f2902bdc640 1 -- 172.21.15.156:0/1813714338 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28fc162740 msgr2=0x7f28fc1503d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.634+0000 7f2902bdc640 1 --2- 172.21.15.156:0/1813714338 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28fc162740 0x7f28fc1503d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28f400efc0 tx=0x7f28f4009f90 comp rx=0 tx=0).stop 2024-04-24T23:48:31.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.634+0000 7f2902bdc640 1 -- 172.21.15.156:0/1813714338 shutdown_connections 2024-04-24T23:48:31.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.634+0000 7f2902bdc640 1 --2- 172.21.15.156:0/1813714338 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28fc162740 0x7f28fc1503d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.634+0000 7f2902bdc640 1 --2- 172.21.15.156:0/1813714338 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28fc0b5500 0x7f28fc15c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.634+0000 7f2902bdc640 1 --2- 172.21.15.156:0/1813714338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28fc0ae4c0 0x7f28fc0b15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.634 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.634+0000 7f2902bdc640 1 -- 172.21.15.156:0/1813714338 >> 172.21.15.156:0/1813714338 conn(0x7f28fc0b2910 msgr2=0x7f28fc0cd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:31.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.634+0000 7f2902bdc640 1 -- 172.21.15.156:0/1813714338 shutdown_connections 2024-04-24T23:48:31.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.634+0000 7f2902bdc640 1 -- 172.21.15.156:0/1813714338 wait complete. 2024-04-24T23:48:31.645 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 5624c321-08e4-4673-9bdb-9b657d33a93e 1 2024-04-24T23:48:31.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.742+0000 7f64774b3640 1 Processor -- start 2024-04-24T23:48:31.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.743+0000 7f64774b3640 1 -- start start 2024-04-24T23:48:31.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.743+0000 7f64774b3640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f64700cf0f0 0x7f64700d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:31.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.743+0000 7f64774b3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64700cf5d0 con 0x7f647015efe0 2024-04-24T23:48:31.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.743+0000 7f64774b3640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64700cf8b0 con 0x7f6470143e80 2024-04-24T23:48:31.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.743+0000 7f64774b3640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f64700d01e0 con 0x7f64700cf0f0 2024-04-24T23:48:31.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.744+0000 7f647524f640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f6470143e80 0x7f647015e8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:55446/0 (socket says 172.21.15.156:55446) 2024-04-24T23:48:31.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.744+0000 7f647524f640 1 -- 172.21.15.156:0/3252817724 learned_addr learned my addr 172.21.15.156:0/3252817724 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-04-24T23:48:31.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.744+0000 7f6475a50640 1 --2- 172.21.15.156:0/3252817724 >> v2:172.21.15.156:3300/0 conn(0x7f64700cf0f0 0x7f64700d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:31.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.744+0000 7f6475a50640 1 -- 172.21.15.156:0/3252817724 >> v1:172.21.15.156:6789/0 conn(0x7f6470143e80 legacy=0x7f647015e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.744+0000 7f6475a50640 1 -- 172.21.15.156:0/3252817724 >> v1:172.21.15.16:6789/0 conn(0x7f647015efe0 legacy=0x7f64700b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.744+0000 7f6475a50640 1 -- 172.21.15.156:0/3252817724 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64700cfb90 con 0x7f64700cf0f0 2024-04-24T23:48:31.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.745+0000 7f6475a50640 1 --2- 172.21.15.156:0/3252817724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64700cf0f0 0x7f64700d0670 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6468009d90 tx=0x7f646802a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=deb462962e9af7ef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:31.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.745+0000 7f6467fff640 1 -- 172.21.15.156:0/3252817724 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6468007740 con 0x7f64700cf0f0 2024-04-24T23:48:31.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.745+0000 7f6467fff640 1 -- 172.21.15.156:0/3252817724 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64680078e0 con 0x7f64700cf0f0 2024-04-24T23:48:31.745 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.745+0000 7f6467fff640 1 -- 172.21.15.156:0/3252817724 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6468007cb0 con 0x7f64700cf0f0 2024-04-24T23:48:31.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.746+0000 7f64774b3640 1 -- 172.21.15.156:0/3252817724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64700cf0f0 msgr2=0x7f64700d0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.746+0000 7f64774b3640 1 --2- 172.21.15.156:0/3252817724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64700cf0f0 0x7f64700d0670 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6468009d90 tx=0x7f646802a6f0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.746+0000 7f64774b3640 1 -- 172.21.15.156:0/3252817724 shutdown_connections 2024-04-24T23:48:31.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.746+0000 7f64774b3640 1 --2- 172.21.15.156:0/3252817724 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64700cf0f0 0x7f64700d0670 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.746+0000 7f64774b3640 1 -- 172.21.15.156:0/3252817724 >> 172.21.15.156:0/3252817724 conn(0x7f64700b29b0 msgr2=0x7f64700b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:31.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.746+0000 7f64774b3640 1 -- 172.21.15.156:0/3252817724 shutdown_connections 2024-04-24T23:48:31.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.746+0000 7f64774b3640 1 -- 172.21.15.156:0/3252817724 wait complete. 2024-04-24T23:48:31.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.747+0000 7f64774b3640 1 Processor -- start 2024-04-24T23:48:31.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.747+0000 7f64774b3640 1 -- start start 2024-04-24T23:48:31.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f64774b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64700cf0f0 0x7f647006b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:31.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f64774b3640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6470143e80 0x7f647006beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:31.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f64774b3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f647015efe0 0x7f6470135d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:31.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f64774b3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f64701443e0 con 0x7f64700cf0f0 2024-04-24T23:48:31.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f64774b3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6470144260 con 0x7f647015efe0 2024-04-24T23:48:31.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f64774b3640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6470144760 con 0x7f6470143e80 2024-04-24T23:48:31.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f6474a4e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6470143e80 0x7f647006beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:31.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f6475a50640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f647015efe0 0x7f6470135d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:31.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f6475a50640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f647015efe0 0x7f6470135d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:54976/0 (socket says 172.21.15.156:54976) 2024-04-24T23:48:31.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f6475a50640 1 -- 172.21.15.156:0/633614005 learned_addr learned my addr 172.21.15.156:0/633614005 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-24T23:48:31.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f6474a4e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6470143e80 0x7f647006beb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:46562/0 (socket says 172.21.15.156:46562) 2024-04-24T23:48:31.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f647524f640 1 --2- 172.21.15.156:0/633614005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64700cf0f0 0x7f647006b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:31.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f6474a4e640 1 -- 172.21.15.156:0/633614005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f647015efe0 msgr2=0x7f6470135d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f6474a4e640 1 --2- 172.21.15.156:0/633614005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f647015efe0 0x7f6470135d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f6474a4e640 1 -- 172.21.15.156:0/633614005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64700cf0f0 msgr2=0x7f647006b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f6474a4e640 1 --2- 172.21.15.156:0/633614005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64700cf0f0 0x7f647006b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f6474a4e640 1 -- 172.21.15.156:0/633614005 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64701377c0 con 0x7f6470143e80 2024-04-24T23:48:31.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f6475a50640 1 --2- 172.21.15.156:0/633614005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f647015efe0 0x7f6470135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:31.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f6474a4e640 1 --2- 172.21.15.156:0/633614005 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6470143e80 0x7f647006beb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f646000c3c0 tx=0x7f646000c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:31.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.748+0000 7f6465ffb640 1 -- 172.21.15.156:0/633614005 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64600052d0 con 0x7f6470143e80 2024-04-24T23:48:31.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.749+0000 7f6465ffb640 1 -- 172.21.15.156:0/633614005 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6460005490 con 0x7f6470143e80 2024-04-24T23:48:31.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.749+0000 7f6465ffb640 1 -- 172.21.15.156:0/633614005 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460007260 con 0x7f6470143e80 2024-04-24T23:48:31.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.749+0000 7f64774b3640 1 -- 172.21.15.156:0/633614005 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6470137aa0 con 0x7f6470143e80 2024-04-24T23:48:31.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.749+0000 7f6465ffb640 1 -- 172.21.15.156:0/633614005 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6460007400 con 0x7f6470143e80 2024-04-24T23:48:31.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.749+0000 7f64774b3640 1 -- 172.21.15.156:0/633614005 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6470138060 con 0x7f6470143e80 2024-04-24T23:48:31.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.749+0000 7f6465ffb640 1 -- 172.21.15.156:0/633614005 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f6460003650 con 0x7f6470143e80 2024-04-24T23:48:31.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.750+0000 7f64774b3640 1 -- 172.21.15.156:0/633614005 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6470068f90 con 0x7f6470143e80 2024-04-24T23:48:31.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.752+0000 7f6465ffb640 1 -- 172.21.15.156:0/633614005 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6460021070 con 0x7f6470143e80 2024-04-24T23:48:31.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.823+0000 7f64774b3640 1 -- 172.21.15.156:0/633614005 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5624c321-08e4-4673-9bdb-9b657d33a93e", "id": 1} v 0) -- 0x7f6470073370 con 0x7f6470143e80 2024-04-24T23:48:31.825 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.826+0000 7f6465ffb640 1 -- 172.21.15.156:0/633614005 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5624c321-08e4-4673-9bdb-9b657d33a93e", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f6460003b40 con 0x7f6470143e80 2024-04-24T23:48:31.825 INFO:teuthology.orchestra.run.smithi156.stdout:1 2024-04-24T23:48:31.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.827+0000 7f64774b3640 1 -- 172.21.15.156:0/633614005 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6470143e80 msgr2=0x7f647006beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.827+0000 7f64774b3640 1 --2- 172.21.15.156:0/633614005 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6470143e80 0x7f647006beb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f646000c3c0 tx=0x7f646000c880 comp rx=0 tx=0).stop 2024-04-24T23:48:31.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.827+0000 7f64774b3640 1 -- 172.21.15.156:0/633614005 shutdown_connections 2024-04-24T23:48:31.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.827+0000 7f64774b3640 1 --2- 172.21.15.156:0/633614005 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f647015efe0 0x7f6470135d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.827+0000 7f64774b3640 1 --2- 172.21.15.156:0/633614005 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6470143e80 0x7f647006beb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.827+0000 7f64774b3640 1 --2- 172.21.15.156:0/633614005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64700cf0f0 0x7f647006b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.827+0000 7f64774b3640 1 -- 172.21.15.156:0/633614005 >> 172.21.15.156:0/633614005 conn(0x7f64700b29b0 msgr2=0x7f647015b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:31.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.827+0000 7f64774b3640 1 -- 172.21.15.156:0/633614005 shutdown_connections 2024-04-24T23:48:31.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.827+0000 7f64774b3640 1 -- 172.21.15.156:0/633614005 wait complete. 2024-04-24T23:48:31.837 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162 2 2024-04-24T23:48:31.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.937+0000 7f839a516640 1 Processor -- start 2024-04-24T23:48:31.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.937+0000 7f839a516640 1 -- start start 2024-04-24T23:48:31.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.938+0000 7f839a516640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f839415efe0 0x7f83940d0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:31.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.938+0000 7f839a516640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f83940cf0f0 0x7f839406df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:31.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.938+0000 7f839a516640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83940cf5d0 con 0x7f8394143e80 2024-04-24T23:48:31.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.938+0000 7f839a516640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f83940d01e0 con 0x7f83940cf0f0 2024-04-24T23:48:31.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.938+0000 7f839a516640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f83940cf8b0 con 0x7f839415efe0 2024-04-24T23:48:31.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.938+0000 7f83937fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f839415efe0 0x7f83940d0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:31.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.938+0000 7f83937fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f839415efe0 0x7f83940d0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:46564/0 (socket says 172.21.15.156:46564) 2024-04-24T23:48:31.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.938+0000 7f8398ab3640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f83940cf0f0 0x7f839406df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:31.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.938+0000 7f83937fe640 1 -- 172.21.15.156:0/2905528692 learned_addr learned my addr 172.21.15.156:0/2905528692 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-24T23:48:31.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.938+0000 7f8398ab3640 1 -- 172.21.15.156:0/2905528692 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f839415efe0 msgr2=0x7f83940d0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:31.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.938+0000 7f8398ab3640 1 --2- 172.21.15.156:0/2905528692 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f839415efe0 0x7f83940d0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.938+0000 7f8398ab3640 1 -- 172.21.15.156:0/2905528692 >> v1:172.21.15.16:6789/0 conn(0x7f8394143e80 legacy=0x7f839415e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.939+0000 7f8398ab3640 1 -- 172.21.15.156:0/2905528692 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83940cfb90 con 0x7f83940cf0f0 2024-04-24T23:48:31.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.938+0000 7f8392ffd640 1 -- 172.21.15.156:0/2905528692 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1967977107 0 0) 0x7f83940cf5d0 con 0x7f8394143e80 2024-04-24T23:48:31.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.939+0000 7f83937fe640 1 --2- 172.21.15.156:0/2905528692 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f839415efe0 0x7f83940d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:31.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.939+0000 7f8398ab3640 1 --2- 172.21.15.156:0/2905528692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83940cf0f0 0x7f839406df60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f838c009d90 tx=0x7f838c02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=78f3b3362b287591 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:31.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.939+0000 7f8392ffd640 1 -- 172.21.15.156:0/2905528692 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c007740 con 0x7f83940cf0f0 2024-04-24T23:48:31.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.939+0000 7f8392ffd640 1 -- 172.21.15.156:0/2905528692 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f838c0078e0 con 0x7f83940cf0f0 2024-04-24T23:48:31.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.939+0000 7f8392ffd640 1 -- 172.21.15.156:0/2905528692 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c007cb0 con 0x7f83940cf0f0 2024-04-24T23:48:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.940+0000 7f839a516640 1 -- 172.21.15.156:0/2905528692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83940cf0f0 msgr2=0x7f839406df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.940+0000 7f839a516640 1 --2- 172.21.15.156:0/2905528692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83940cf0f0 0x7f839406df60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f838c009d90 tx=0x7f838c02a660 comp rx=0 tx=0).stop 2024-04-24T23:48:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.940+0000 7f839a516640 1 -- 172.21.15.156:0/2905528692 shutdown_connections 2024-04-24T23:48:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.940+0000 7f839a516640 1 --2- 172.21.15.156:0/2905528692 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f83940cf0f0 0x7f839406df60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.940+0000 7f839a516640 1 --2- 172.21.15.156:0/2905528692 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f839415efe0 0x7f83940d0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.940+0000 7f839a516640 1 -- 172.21.15.156:0/2905528692 >> 172.21.15.156:0/2905528692 conn(0x7f83940b29b0 msgr2=0x7f83940b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.940+0000 7f839a516640 1 -- 172.21.15.156:0/2905528692 shutdown_connections 2024-04-24T23:48:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.940+0000 7f839a516640 1 -- 172.21.15.156:0/2905528692 wait complete. 2024-04-24T23:48:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.941+0000 7f839a516640 1 Processor -- start 2024-04-24T23:48:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.942+0000 7f839a516640 1 -- start start 2024-04-24T23:48:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.942+0000 7f839a516640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83940cf0f0 0x7f8394154490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.942+0000 7f839a516640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8394143e80 0x7f839413fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.942+0000 7f839a516640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f839415efe0 0x7f8394140180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.942+0000 7f839a516640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83941443e0 con 0x7f8394143e80 2024-04-24T23:48:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.942+0000 7f839a516640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8394144260 con 0x7f839415efe0 2024-04-24T23:48:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.942+0000 7f839a516640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8394144760 con 0x7f83940cf0f0 2024-04-24T23:48:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.942+0000 7f8398ab3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f839415efe0 0x7f8394140180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.942+0000 7f8393fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83940cf0f0 0x7f8394154490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.942+0000 7f8398ab3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f839415efe0 0x7f8394140180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:54988/0 (socket says 172.21.15.156:54988) 2024-04-24T23:48:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.942+0000 7f8398ab3640 1 -- 172.21.15.156:0/855363352 learned_addr learned my addr 172.21.15.156:0/855363352 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-24T23:48:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.942+0000 7f8393fff640 1 --2- 172.21.15.156:0/855363352 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83940cf0f0 0x7f8394154490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:46580/0 (socket says 172.21.15.156:46580) 2024-04-24T23:48:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.942+0000 7f83937fe640 1 --2- 172.21.15.156:0/855363352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8394143e80 0x7f839413fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:31.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.943+0000 7f8398ab3640 1 -- 172.21.15.156:0/855363352 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83940cf0f0 msgr2=0x7f8394154490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.943+0000 7f8398ab3640 1 --2- 172.21.15.156:0/855363352 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83940cf0f0 0x7f8394154490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.943+0000 7f8398ab3640 1 -- 172.21.15.156:0/855363352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8394143e80 msgr2=0x7f839413fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:31.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.943+0000 7f8398ab3640 1 --2- 172.21.15.156:0/855363352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8394143e80 0x7f839413fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:31.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.943+0000 7f8398ab3640 1 -- 172.21.15.156:0/855363352 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83941407e0 con 0x7f839415efe0 2024-04-24T23:48:31.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.943+0000 7f83937fe640 1 --2- 172.21.15.156:0/855363352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8394143e80 0x7f839413fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:31.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.943+0000 7f8398ab3640 1 --2- 172.21.15.156:0/855363352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f839415efe0 0x7f8394140180 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f838c02ac50 tx=0x7f838c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:31.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.943+0000 7f8390ff9640 1 -- 172.21.15.156:0/855363352 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c040070 con 0x7f839415efe0 2024-04-24T23:48:31.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.943+0000 7f8390ff9640 1 -- 172.21.15.156:0/855363352 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f838c031bc0 con 0x7f839415efe0 2024-04-24T23:48:31.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.943+0000 7f8390ff9640 1 -- 172.21.15.156:0/855363352 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f838c009450 con 0x7f839415efe0 2024-04-24T23:48:31.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.943+0000 7f839a516640 1 -- 172.21.15.156:0/855363352 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8394140ac0 con 0x7f839415efe0 2024-04-24T23:48:31.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.943+0000 7f839a516640 1 -- 172.21.15.156:0/855363352 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8394141520 con 0x7f839415efe0 2024-04-24T23:48:31.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.943+0000 7f8390ff9640 1 -- 172.21.15.156:0/855363352 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f838c039040 con 0x7f839415efe0 2024-04-24T23:48:31.945 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.943+0000 7f8390ff9640 1 -- 172.21.15.156:0/855363352 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f838c045d20 con 0x7f839415efe0 2024-04-24T23:48:31.946 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.944+0000 7f839a516640 1 -- 172.21.15.156:0/855363352 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8394068f90 con 0x7f839415efe0 2024-04-24T23:48:31.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:31.948+0000 7f8390ff9640 1 -- 172.21.15.156:0/855363352 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f838c053080 con 0x7f839415efe0 2024-04-24T23:48:32.021 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.021+0000 7f839a516640 1 -- 172.21.15.156:0/855363352 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162", "id": 2} v 0) -- 0x7f8394073370 con 0x7f839415efe0 2024-04-24T23:48:32.023 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.024+0000 7f8390ff9640 1 -- 172.21.15.156:0/855363352 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f838c007840 con 0x7f839415efe0 2024-04-24T23:48:32.023 INFO:teuthology.orchestra.run.smithi156.stdout:2 2024-04-24T23:48:32.024 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.025+0000 7f839a516640 1 -- 172.21.15.156:0/855363352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f839415efe0 msgr2=0x7f8394140180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.025 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.025+0000 7f839a516640 1 --2- 172.21.15.156:0/855363352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f839415efe0 0x7f8394140180 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f838c02ac50 tx=0x7f838c004a40 comp rx=0 tx=0).stop 2024-04-24T23:48:32.025 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.025+0000 7f839a516640 1 -- 172.21.15.156:0/855363352 shutdown_connections 2024-04-24T23:48:32.025 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.025+0000 7f839a516640 1 --2- 172.21.15.156:0/855363352 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f839415efe0 0x7f8394140180 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.025 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.025+0000 7f839a516640 1 --2- 172.21.15.156:0/855363352 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8394143e80 0x7f839413fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.025 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.025+0000 7f839a516640 1 --2- 172.21.15.156:0/855363352 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83940cf0f0 0x7f8394154490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.025 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.025+0000 7f839a516640 1 -- 172.21.15.156:0/855363352 >> 172.21.15.156:0/855363352 conn(0x7f83940b29b0 msgr2=0x7f839415b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:32.025 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.025+0000 7f839a516640 1 -- 172.21.15.156:0/855363352 shutdown_connections 2024-04-24T23:48:32.025 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.025+0000 7f839a516640 1 -- 172.21.15.156:0/855363352 wait complete. 2024-04-24T23:48:32.035 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 72dbaef3-46da-45b1-9744-d8abb632614c 3 2024-04-24T23:48:32.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.132+0000 7f91818f7640 1 Processor -- start 2024-04-24T23:48:32.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.132+0000 7f91818f7640 1 -- start start 2024-04-24T23:48:32.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.133+0000 7f91818f7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f917c0b43c0 0x7f917c168e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.133+0000 7f91818f7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f917c15e300 con 0x7f917c15df20 2024-04-24T23:48:32.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.133+0000 7f91818f7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f917c0af530 con 0x7f917c0af0a0 2024-04-24T23:48:32.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.133+0000 7f91818f7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f917c0d00e0 con 0x7f917c0b43c0 2024-04-24T23:48:32.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.133+0000 7f917a7fc640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f917c0af0a0 0x7f917c0b3cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:55460/0 (socket says 172.21.15.156:55460) 2024-04-24T23:48:32.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.133+0000 7f917a7fc640 1 -- 172.21.15.156:0/2047342248 learned_addr learned my addr 172.21.15.156:0/2047342248 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-04-24T23:48:32.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.133+0000 7f917b7fe640 1 --2- 172.21.15.156:0/2047342248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f917c0b43c0 0x7f917c168e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.133+0000 7f9179ffb640 1 -- 172.21.15.156:0/2047342248 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1106385169 0 0) 0x7f917c0af530 con 0x7f917c0af0a0 2024-04-24T23:48:32.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.134+0000 7f917b7fe640 1 -- 172.21.15.156:0/2047342248 >> v1:172.21.15.156:6789/0 conn(0x7f917c0af0a0 legacy=0x7f917c0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.134+0000 7f917b7fe640 1 -- 172.21.15.156:0/2047342248 >> v1:172.21.15.16:6789/0 conn(0x7f917c15df20 legacy=0x7f917c0cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.134 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.134+0000 7f917b7fe640 1 -- 172.21.15.156:0/2047342248 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917c0d0660 con 0x7f917c0b43c0 2024-04-24T23:48:32.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.134+0000 7f917b7fe640 1 --2- 172.21.15.156:0/2047342248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f917c0b43c0 0x7f917c168e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9174009dc0 tx=0x7f917402a5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50baa1aed6b0a2fe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:32.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.134+0000 7f9179ffb640 1 -- 172.21.15.156:0/2047342248 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9174007740 con 0x7f917c0b43c0 2024-04-24T23:48:32.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.134+0000 7f9179ffb640 1 -- 172.21.15.156:0/2047342248 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91740078e0 con 0x7f917c0b43c0 2024-04-24T23:48:32.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.134+0000 7f9179ffb640 1 -- 172.21.15.156:0/2047342248 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9174007cb0 con 0x7f917c0b43c0 2024-04-24T23:48:32.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.135+0000 7f91818f7640 1 -- 172.21.15.156:0/2047342248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f917c0b43c0 msgr2=0x7f917c168e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.135+0000 7f91818f7640 1 --2- 172.21.15.156:0/2047342248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f917c0b43c0 0x7f917c168e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9174009dc0 tx=0x7f917402a5b0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.135+0000 7f91818f7640 1 -- 172.21.15.156:0/2047342248 shutdown_connections 2024-04-24T23:48:32.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.135+0000 7f91818f7640 1 --2- 172.21.15.156:0/2047342248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f917c0b43c0 0x7f917c168e40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.135+0000 7f91818f7640 1 -- 172.21.15.156:0/2047342248 >> 172.21.15.156:0/2047342248 conn(0x7f917c0b29b0 msgr2=0x7f917c0b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:32.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.135+0000 7f91818f7640 1 -- 172.21.15.156:0/2047342248 shutdown_connections 2024-04-24T23:48:32.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.135+0000 7f91818f7640 1 -- 172.21.15.156:0/2047342248 wait complete. 2024-04-24T23:48:32.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.136+0000 7f91818f7640 1 Processor -- start 2024-04-24T23:48:32.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.136+0000 7f91818f7640 1 -- start start 2024-04-24T23:48:32.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.137+0000 7f91818f7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f917c0af0a0 0x7f917c14c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.137+0000 7f91818f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f917c0b43c0 0x7f917c15efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.137+0000 7f91818f7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f917c15df20 0x7f917c072070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.137+0000 7f91818f7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f917c0cf4b0 con 0x7f917c0b43c0 2024-04-24T23:48:32.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.137+0000 7f91818f7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f917c0cf1d0 con 0x7f917c0af0a0 2024-04-24T23:48:32.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.137+0000 7f91818f7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f917c0d0940 con 0x7f917c15df20 2024-04-24T23:48:32.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.137+0000 7f917affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f917c0af0a0 0x7f917c14c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.137+0000 7f917b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f917c15df20 0x7f917c072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.138 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.137+0000 7f917affd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f917c0af0a0 0x7f917c14c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55000/0 (socket says 172.21.15.156:55000) 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.137+0000 7f917b7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f917c15df20 0x7f917c072070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:46602/0 (socket says 172.21.15.156:46602) 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.137+0000 7f917affd640 1 -- 172.21.15.156:0/1100954042 learned_addr learned my addr 172.21.15.156:0/1100954042 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.137+0000 7f917a7fc640 1 --2- 172.21.15.156:0/1100954042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f917c0b43c0 0x7f917c15efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.138+0000 7f917b7fe640 1 -- 172.21.15.156:0/1100954042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f917c0af0a0 msgr2=0x7f917c14c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.138+0000 7f917b7fe640 1 --2- 172.21.15.156:0/1100954042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f917c0af0a0 0x7f917c14c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.138+0000 7f917b7fe640 1 -- 172.21.15.156:0/1100954042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f917c0b43c0 msgr2=0x7f917c15efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.138+0000 7f917b7fe640 1 --2- 172.21.15.156:0/1100954042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f917c0b43c0 0x7f917c15efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.138+0000 7f917b7fe640 1 -- 172.21.15.156:0/1100954042 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f917c06baa0 con 0x7f917c15df20 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.138+0000 7f917b7fe640 1 --2- 172.21.15.156:0/1100954042 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f917c15df20 0x7f917c072070 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9174031000 tx=0x7f9174004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.138+0000 7f9163fff640 1 -- 172.21.15.156:0/1100954042 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f917403f070 con 0x7f917c15df20 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.138+0000 7f9163fff640 1 -- 172.21.15.156:0/1100954042 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9174031bc0 con 0x7f917c15df20 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.138+0000 7f91818f7640 1 -- 172.21.15.156:0/1100954042 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f917c06bd40 con 0x7f917c15df20 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.138+0000 7f9163fff640 1 -- 172.21.15.156:0/1100954042 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9174044460 con 0x7f917c15df20 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.138+0000 7f91818f7640 1 -- 172.21.15.156:0/1100954042 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f917c06c300 con 0x7f917c15df20 2024-04-24T23:48:32.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.138+0000 7f9163fff640 1 -- 172.21.15.156:0/1100954042 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9174044600 con 0x7f917c15df20 2024-04-24T23:48:32.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.138+0000 7f9163fff640 1 -- 172.21.15.156:0/1100954042 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f917403b9e0 con 0x7f917c15df20 2024-04-24T23:48:32.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.139+0000 7f91818f7640 1 -- 172.21.15.156:0/1100954042 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f917c068f90 con 0x7f917c15df20 2024-04-24T23:48:32.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.141+0000 7f9163fff640 1 -- 172.21.15.156:0/1100954042 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9174052080 con 0x7f917c15df20 2024-04-24T23:48:32.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.213+0000 7f91818f7640 1 -- 172.21.15.156:0/1100954042 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd new", "uuid": "72dbaef3-46da-45b1-9744-d8abb632614c", "id": 3} v 0) -- 0x7f917c073380 con 0x7f917c15df20 2024-04-24T23:48:32.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.216+0000 7f9163fff640 1 -- 172.21.15.156:0/1100954042 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "72dbaef3-46da-45b1-9744-d8abb632614c", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f917403bc80 con 0x7f917c15df20 2024-04-24T23:48:32.215 INFO:teuthology.orchestra.run.smithi156.stdout:3 2024-04-24T23:48:32.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.217+0000 7f91818f7640 1 -- 172.21.15.156:0/1100954042 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f917c15df20 msgr2=0x7f917c072070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.217+0000 7f91818f7640 1 --2- 172.21.15.156:0/1100954042 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f917c15df20 0x7f917c072070 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9174031000 tx=0x7f9174004a40 comp rx=0 tx=0).stop 2024-04-24T23:48:32.216 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.217+0000 7f91818f7640 1 -- 172.21.15.156:0/1100954042 shutdown_connections 2024-04-24T23:48:32.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.217+0000 7f91818f7640 1 --2- 172.21.15.156:0/1100954042 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f917c15df20 0x7f917c072070 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.217+0000 7f91818f7640 1 --2- 172.21.15.156:0/1100954042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f917c0b43c0 0x7f917c15efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.217+0000 7f91818f7640 1 --2- 172.21.15.156:0/1100954042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f917c0af0a0 0x7f917c14c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.217+0000 7f91818f7640 1 -- 172.21.15.156:0/1100954042 >> 172.21.15.156:0/1100954042 conn(0x7f917c0b29b0 msgr2=0x7f917c1570e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:32.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.217+0000 7f91818f7640 1 -- 172.21.15.156:0/1100954042 shutdown_connections 2024-04-24T23:48:32.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.217+0000 7f91818f7640 1 -- 172.21.15.156:0/1100954042 wait complete. 2024-04-24T23:48:32.227 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 75482b46-cd25-4196-8026-87fac8e73a41 4 2024-04-24T23:48:32.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.324+0000 7f5799a9e640 1 Processor -- start 2024-04-24T23:48:32.324 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.324+0000 7f5799a9e640 1 -- start start 2024-04-24T23:48:32.325 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.325+0000 7f5799a9e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f579415dfd0 0x7f579406b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.325+0000 7f5799a9e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57940b8060 0x7f579406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.325+0000 7f5799a9e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57940d24d0 con 0x7f579406bc40 2024-04-24T23:48:32.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.325+0000 7f5799a9e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f57940d27b0 con 0x7f579415dfd0 2024-04-24T23:48:32.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.325+0000 7f5799a9e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f57940d2a90 con 0x7f57940b8060 2024-04-24T23:48:32.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.325+0000 7f57937fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f579415dfd0 0x7f579406b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.325+0000 7f5793fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57940b8060 0x7f579406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.325+0000 7f57937fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f579415dfd0 0x7f579406b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55012/0 (socket says 172.21.15.156:55012) 2024-04-24T23:48:32.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.325+0000 7f57937fe640 1 -- 172.21.15.156:0/3837747035 learned_addr learned my addr 172.21.15.156:0/3837747035 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-24T23:48:32.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.325+0000 7f5793fff640 1 -- 172.21.15.156:0/3837747035 >> v2:172.21.15.156:3300/0 conn(0x7f579415dfd0 msgr2=0x7f579406b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:32.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.325+0000 7f5793fff640 1 --2- 172.21.15.156:0/3837747035 >> v2:172.21.15.156:3300/0 conn(0x7f579415dfd0 0x7f579406b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.325+0000 7f5793fff640 1 -- 172.21.15.156:0/3837747035 >> v1:172.21.15.156:6789/0 conn(0x7f579406bc40 legacy=0x7f57940cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.326 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.325+0000 7f5793fff640 1 -- 172.21.15.156:0/3837747035 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57940d2d70 con 0x7f57940b8060 2024-04-24T23:48:32.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.325+0000 7f57937fe640 1 --2- 172.21.15.156:0/3837747035 >> v2:172.21.15.156:3300/0 conn(0x7f579415dfd0 0x7f579406b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:32.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.326+0000 7f5793fff640 1 --2- 172.21.15.156:0/3837747035 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57940b8060 0x7f579406c0d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f578c009d90 tx=0x7f578c02a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d60cc98cead8d5cf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:32.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.326+0000 7f5792ffd640 1 -- 172.21.15.156:0/3837747035 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578c031600 con 0x7f57940b8060 2024-04-24T23:48:32.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.326+0000 7f5792ffd640 1 -- 172.21.15.156:0/3837747035 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f578c0317a0 con 0x7f57940b8060 2024-04-24T23:48:32.327 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.326+0000 7f5792ffd640 1 -- 172.21.15.156:0/3837747035 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f578c031b70 con 0x7f57940b8060 2024-04-24T23:48:32.327 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:32.325+0000 7fac8b91c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T23:48:32.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.327+0000 7f5799a9e640 1 -- 172.21.15.156:0/3837747035 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57940b8060 msgr2=0x7f579406c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.327+0000 7f5799a9e640 1 --2- 172.21.15.156:0/3837747035 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57940b8060 0x7f579406c0d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f578c009d90 tx=0x7f578c02a5d0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.327+0000 7f5799a9e640 1 -- 172.21.15.156:0/3837747035 shutdown_connections 2024-04-24T23:48:32.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.327+0000 7f5799a9e640 1 --2- 172.21.15.156:0/3837747035 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57940b8060 0x7f579406c0d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.327+0000 7f5799a9e640 1 --2- 172.21.15.156:0/3837747035 >> v2:172.21.15.156:3300/0 conn(0x7f579415dfd0 0x7f579406b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.330 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.327+0000 7f5799a9e640 1 -- 172.21.15.156:0/3837747035 >> 172.21.15.156:0/3837747035 conn(0x7f57940d3dc0 msgr2=0x7f57940d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.327+0000 7f5799a9e640 1 -- 172.21.15.156:0/3837747035 shutdown_connections 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.327+0000 7f5799a9e640 1 -- 172.21.15.156:0/3837747035 wait complete. 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.328+0000 7f5799a9e640 1 Processor -- start 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.328+0000 7f5799a9e640 1 -- start start 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f5799a9e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f579406bc40 0x7f57940b2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f5799a9e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57940b8060 0x7f57940cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f5799a9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f579415dfd0 0x7f57941f7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f5799a9e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f579406b830 con 0x7f579415dfd0 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f5799a9e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f57940cead0 con 0x7f579406bc40 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f5799a9e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f579406b9b0 con 0x7f57940b8060 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f578bfff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57940b8060 0x7f57940cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f57937fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f579406bc40 0x7f57940b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f578bfff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57940b8060 0x7f57940cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:46626/0 (socket says 172.21.15.156:46626) 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f578bfff640 1 -- 172.21.15.156:0/1325840284 learned_addr learned my addr 172.21.15.156:0/1325840284 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-24T23:48:32.331 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f5793fff640 1 --2- 172.21.15.156:0/1325840284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f579415dfd0 0x7f57941f7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f578bfff640 1 -- 172.21.15.156:0/1325840284 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f579406bc40 msgr2=0x7f57940b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f578bfff640 1 --2- 172.21.15.156:0/1325840284 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f579406bc40 0x7f57940b2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f578bfff640 1 -- 172.21.15.156:0/1325840284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f579415dfd0 msgr2=0x7f57941f7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f578bfff640 1 --2- 172.21.15.156:0/1325840284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f579415dfd0 0x7f57941f7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.332 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f578bfff640 1 -- 172.21.15.156:0/1325840284 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57940ce1b0 con 0x7f57940b8060 2024-04-24T23:48:32.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f57937fe640 1 --2- 172.21.15.156:0/1325840284 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f579406bc40 0x7f57940b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:32.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.329+0000 7f578bfff640 1 --2- 172.21.15.156:0/1325840284 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57940b8060 0x7f57940cd100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f577c00c400 tx=0x7f577c00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:32.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.330+0000 7f5790ff9640 1 -- 172.21.15.156:0/1325840284 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c0052d0 con 0x7f57940b8060 2024-04-24T23:48:32.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.330+0000 7f5790ff9640 1 -- 172.21.15.156:0/1325840284 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f577c005490 con 0x7f57940b8060 2024-04-24T23:48:32.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.330+0000 7f5790ff9640 1 -- 172.21.15.156:0/1325840284 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f577c007260 con 0x7f57940b8060 2024-04-24T23:48:32.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.330+0000 7f5799a9e640 1 -- 172.21.15.156:0/1325840284 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f579414a1a0 con 0x7f57940b8060 2024-04-24T23:48:32.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.330+0000 7f5799a9e640 1 -- 172.21.15.156:0/1325840284 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f57940cdbf0 con 0x7f57940b8060 2024-04-24T23:48:32.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.330+0000 7f5790ff9640 1 -- 172.21.15.156:0/1325840284 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f577c007400 con 0x7f57940b8060 2024-04-24T23:48:32.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.330+0000 7f5790ff9640 1 -- 172.21.15.156:0/1325840284 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f577c011590 con 0x7f57940b8060 2024-04-24T23:48:32.333 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.331+0000 7f5799a9e640 1 -- 172.21.15.156:0/1325840284 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5794068f90 con 0x7f57940b8060 2024-04-24T23:48:32.334 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.334+0000 7f5790ff9640 1 -- 172.21.15.156:0/1325840284 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f577c0067d0 con 0x7f57940b8060 2024-04-24T23:48:32.343 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:32.344+0000 7fcfc9212040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T23:48:32.404 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.405+0000 7f5799a9e640 1 -- 172.21.15.156:0/1325840284 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd new", "uuid": "75482b46-cd25-4196-8026-87fac8e73a41", "id": 4} v 0) -- 0x7f5794067730 con 0x7f57940b8060 2024-04-24T23:48:32.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.408+0000 7f5790ff9640 1 -- 172.21.15.156:0/1325840284 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "75482b46-cd25-4196-8026-87fac8e73a41", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f577c0035f0 con 0x7f57940b8060 2024-04-24T23:48:32.407 INFO:teuthology.orchestra.run.smithi156.stdout:4 2024-04-24T23:48:32.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.409+0000 7f5799a9e640 1 -- 172.21.15.156:0/1325840284 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57940b8060 msgr2=0x7f57940cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.409+0000 7f5799a9e640 1 --2- 172.21.15.156:0/1325840284 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57940b8060 0x7f57940cd100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f577c00c400 tx=0x7f577c00c8c0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.409+0000 7f5799a9e640 1 -- 172.21.15.156:0/1325840284 shutdown_connections 2024-04-24T23:48:32.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.409+0000 7f5799a9e640 1 --2- 172.21.15.156:0/1325840284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f579415dfd0 0x7f57941f7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.409+0000 7f5799a9e640 1 --2- 172.21.15.156:0/1325840284 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57940b8060 0x7f57940cd100 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.409+0000 7f5799a9e640 1 --2- 172.21.15.156:0/1325840284 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f579406bc40 0x7f57940b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.409+0000 7f5799a9e640 1 -- 172.21.15.156:0/1325840284 >> 172.21.15.156:0/1325840284 conn(0x7f57940d3dc0 msgr2=0x7f57940b47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:32.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.409+0000 7f5799a9e640 1 -- 172.21.15.156:0/1325840284 shutdown_connections 2024-04-24T23:48:32.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.409+0000 7f5799a9e640 1 -- 172.21.15.156:0/1325840284 wait complete. 2024-04-24T23:48:32.416 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:32.416+0000 7fac8b91c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T23:48:32.419 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d 5 2024-04-24T23:48:32.434 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:32.435+0000 7fcfc9212040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T23:48:32.461 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:32.460+0000 7fac8b91c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T23:48:32.480 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:32.480+0000 7fcfc9212040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T23:48:32.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.515+0000 7f9ea0d5c640 1 Processor -- start 2024-04-24T23:48:32.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.515+0000 7f9ea0d5c640 1 -- start start 2024-04-24T23:48:32.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.516+0000 7f9ea0d5c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e9c0d1500 0x7f9e9c0cda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.516+0000 7f9ea0d5c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e9c15d090 0x7f9e9c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.516+0000 7f9ea0d5c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9e9c0d2030 0x7f9e9c0ce570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.516+0000 7f9ea0d5c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9e9c160930 con 0x7f9e9c15d090 2024-04-24T23:48:32.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.516+0000 7f9ea0d5c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f9e9c1604a0 con 0x7f9e9c0d2030 2024-04-24T23:48:32.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.516+0000 7f9ea0d5c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f9e9c0ce330 con 0x7f9e9c0d1500 2024-04-24T23:48:32.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.516+0000 7f9e9a59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e9c0d1500 0x7f9e9c0cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.516+0000 7f9e9ad9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9e9c0d2030 0x7f9e9c0ce570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.517+0000 7f9e9a59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e9c0d1500 0x7f9e9c0cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:46634/0 (socket says 172.21.15.156:46634) 2024-04-24T23:48:32.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.517+0000 7f9e9ad9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9e9c0d2030 0x7f9e9c0ce570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55034/0 (socket says 172.21.15.156:55034) 2024-04-24T23:48:32.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.517+0000 7f9e9a59c640 1 -- 172.21.15.156:0/864399611 learned_addr learned my addr 172.21.15.156:0/864399611 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-24T23:48:32.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.517+0000 7f9e99d9b640 1 --2- 172.21.15.156:0/864399611 >> v2:172.21.15.16:3300/0 conn(0x7f9e9c15d090 0x7f9e9c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.517+0000 7f9e9ad9d640 1 -- 172.21.15.156:0/864399611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e9c0d1500 msgr2=0x7f9e9c0cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:32.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.517+0000 7f9e9ad9d640 1 --2- 172.21.15.156:0/864399611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e9c0d1500 0x7f9e9c0cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.517+0000 7f9e9ad9d640 1 -- 172.21.15.156:0/864399611 >> v2:172.21.15.16:3300/0 conn(0x7f9e9c15d090 msgr2=0x7f9e9c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:32.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.517+0000 7f9e9ad9d640 1 --2- 172.21.15.156:0/864399611 >> v2:172.21.15.16:3300/0 conn(0x7f9e9c15d090 0x7f9e9c0d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.517+0000 7f9e9ad9d640 1 -- 172.21.15.156:0/864399611 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9c0d18e0 con 0x7f9e9c0d2030 2024-04-24T23:48:32.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.517+0000 7f9e9a59c640 1 --2- 172.21.15.156:0/864399611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e9c0d1500 0x7f9e9c0cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:32.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.517+0000 7f9e9ad9d640 1 --2- 172.21.15.156:0/864399611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e9c0d2030 0x7f9e9c0ce570 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0cd1b0 tx=0x7f9e9402a570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e42d99d44ae9af78 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:32.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.518+0000 7f9e9959a640 1 -- 172.21.15.156:0/864399611 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94007740 con 0x7f9e9c0d2030 2024-04-24T23:48:32.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.518+0000 7f9e9959a640 1 -- 172.21.15.156:0/864399611 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e940078e0 con 0x7f9e9c0d2030 2024-04-24T23:48:32.518 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.518+0000 7f9e9959a640 1 -- 172.21.15.156:0/864399611 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e94007cb0 con 0x7f9e9c0d2030 2024-04-24T23:48:32.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.518+0000 7f9ea0d5c640 1 -- 172.21.15.156:0/864399611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e9c0d2030 msgr2=0x7f9e9c0ce570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.518+0000 7f9ea0d5c640 1 --2- 172.21.15.156:0/864399611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e9c0d2030 0x7f9e9c0ce570 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9e9c0cd1b0 tx=0x7f9e9402a570 comp rx=0 tx=0).stop 2024-04-24T23:48:32.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.518+0000 7f9ea0d5c640 1 -- 172.21.15.156:0/864399611 shutdown_connections 2024-04-24T23:48:32.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.518+0000 7f9ea0d5c640 1 --2- 172.21.15.156:0/864399611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e9c0d2030 0x7f9e9c0ce570 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.521 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.518+0000 7f9ea0d5c640 1 --2- 172.21.15.156:0/864399611 >> v2:172.21.15.16:3300/0 conn(0x7f9e9c15d090 0x7f9e9c0d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.518+0000 7f9ea0d5c640 1 --2- 172.21.15.156:0/864399611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e9c0d1500 0x7f9e9c0cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.518+0000 7f9ea0d5c640 1 -- 172.21.15.156:0/864399611 >> 172.21.15.156:0/864399611 conn(0x7f9e9c06b0e0 msgr2=0x7f9e9c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:32.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.518+0000 7f9ea0d5c640 1 -- 172.21.15.156:0/864399611 shutdown_connections 2024-04-24T23:48:32.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.518+0000 7f9ea0d5c640 1 -- 172.21.15.156:0/864399611 wait complete. 2024-04-24T23:48:32.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.520+0000 7f9ea0d5c640 1 Processor -- start 2024-04-24T23:48:32.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.520+0000 7f9ea0d5c640 1 -- start start 2024-04-24T23:48:32.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.520+0000 7f9ea0d5c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e9c0d1500 0x7f9e9c156fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.520+0000 7f9ea0d5c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e9c0d2030 0x7f9e9c1e7d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.520+0000 7f9ea0d5c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e9c15d090 0x7f9e9c0d0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.520+0000 7f9ea0d5c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c160130 con 0x7f9e9c0d1500 2024-04-24T23:48:32.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.520+0000 7f9ea0d5c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9e9c15fe50 con 0x7f9e9c15d090 2024-04-24T23:48:32.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.520+0000 7f9ea0d5c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f9e9c0cdf60 con 0x7f9e9c0d2030 2024-04-24T23:48:32.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e99d9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e9c0d2030 0x7f9e9c1e7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.522 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e9ad9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e9c15d090 0x7f9e9c0d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e99d9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e9c0d2030 0x7f9e9c1e7d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:46650/0 (socket says 172.21.15.156:46650) 2024-04-24T23:48:32.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e99d9b640 1 -- 172.21.15.156:0/2664676224 learned_addr learned my addr 172.21.15.156:0/2664676224 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-24T23:48:32.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e9a59c640 1 --2- 172.21.15.156:0/2664676224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e9c0d1500 0x7f9e9c156fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e99d9b640 1 -- 172.21.15.156:0/2664676224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e9c15d090 msgr2=0x7f9e9c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e99d9b640 1 --2- 172.21.15.156:0/2664676224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e9c15d090 0x7f9e9c0d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e99d9b640 1 -- 172.21.15.156:0/2664676224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e9c0d1500 msgr2=0x7f9e9c156fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e99d9b640 1 --2- 172.21.15.156:0/2664676224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e9c0d1500 0x7f9e9c156fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e99d9b640 1 -- 172.21.15.156:0/2664676224 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e9c1574f0 con 0x7f9e9c0d2030 2024-04-24T23:48:32.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e99d9b640 1 --2- 172.21.15.156:0/2664676224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e9c0d2030 0x7f9e9c1e7d10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400e9b0 tx=0x7f9e8400edb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:32.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e92ffd640 1 -- 172.21.15.156:0/2664676224 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e84017070 con 0x7f9e9c0d2030 2024-04-24T23:48:32.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e92ffd640 1 -- 172.21.15.156:0/2664676224 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e8400c940 con 0x7f9e9c0d2030 2024-04-24T23:48:32.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9ea0d5c640 1 -- 172.21.15.156:0/2664676224 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e9c1e8250 con 0x7f9e9c0d2030 2024-04-24T23:48:32.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e92ffd640 1 -- 172.21.15.156:0/2664676224 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e8400cd30 con 0x7f9e9c0d2030 2024-04-24T23:48:32.523 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9ea0d5c640 1 -- 172.21.15.156:0/2664676224 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e9c0b6440 con 0x7f9e9c0d2030 2024-04-24T23:48:32.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e92ffd640 1 -- 172.21.15.156:0/2664676224 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f9e8401a070 con 0x7f9e9c0d2030 2024-04-24T23:48:32.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.521+0000 7f9e92ffd640 1 -- 172.21.15.156:0/2664676224 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f9e84006570 con 0x7f9e9c0d2030 2024-04-24T23:48:32.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.522+0000 7f9ea0d5c640 1 -- 172.21.15.156:0/2664676224 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e9c068f90 con 0x7f9e9c0d2030 2024-04-24T23:48:32.524 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.524+0000 7f9e92ffd640 1 -- 172.21.15.156:0/2664676224 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f9e9c1574f0 con 0x7f9e9c0d2030 2024-04-24T23:48:32.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.595+0000 7f9ea0d5c640 1 -- 172.21.15.156:0/2664676224 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d", "id": 5} v 0) -- 0x7f9e9c0b2470 con 0x7f9e9c0d2030 2024-04-24T23:48:32.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.598+0000 7f9e92ffd640 1 -- 172.21.15.156:0/2664676224 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f9e84005990 con 0x7f9e9c0d2030 2024-04-24T23:48:32.597 INFO:teuthology.orchestra.run.smithi156.stdout:5 2024-04-24T23:48:32.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.599+0000 7f9ea0d5c640 1 -- 172.21.15.156:0/2664676224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e9c0d2030 msgr2=0x7f9e9c1e7d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.599+0000 7f9ea0d5c640 1 --2- 172.21.15.156:0/2664676224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e9c0d2030 0x7f9e9c1e7d10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9e8400e9b0 tx=0x7f9e8400edb0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.599+0000 7f9ea0d5c640 1 -- 172.21.15.156:0/2664676224 shutdown_connections 2024-04-24T23:48:32.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.599+0000 7f9ea0d5c640 1 --2- 172.21.15.156:0/2664676224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e9c15d090 0x7f9e9c0d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.599+0000 7f9ea0d5c640 1 --2- 172.21.15.156:0/2664676224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e9c0d2030 0x7f9e9c1e7d10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.599+0000 7f9ea0d5c640 1 --2- 172.21.15.156:0/2664676224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e9c0d1500 0x7f9e9c156fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.599+0000 7f9ea0d5c640 1 -- 172.21.15.156:0/2664676224 >> 172.21.15.156:0/2664676224 conn(0x7f9e9c06b0e0 msgr2=0x7f9e9c0b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:32.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.599+0000 7f9ea0d5c640 1 -- 172.21.15.156:0/2664676224 shutdown_connections 2024-04-24T23:48:32.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.599+0000 7f9ea0d5c640 1 -- 172.21.15.156:0/2664676224 wait complete. 2024-04-24T23:48:32.609 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 4f095fbc-1077-4a2d-bde6-98d4dd3b0931 6 2024-04-24T23:48:32.633 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:32.632+0000 7fac8b91c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T23:48:32.653 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:32.654+0000 7fcfc9212040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T23:48:32.682 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:32.681+0000 7fac8b91c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T23:48:32.714 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:32.715+0000 7fcfc9212040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T23:48:32.736 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:32.736+0000 7fac8b91c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T23:48:32.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.747+0000 7f3fcff6a640 1 Processor -- start 2024-04-24T23:48:32.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.747+0000 7f3fcff6a640 1 -- start start 2024-04-24T23:48:32.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.748+0000 7f3fcff6a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fc80b5500 0x7f3fc80d0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.748+0000 7f3fcff6a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fc80b58e0 con 0x7f3fc80ae4c0 2024-04-24T23:48:32.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.748+0000 7f3fcff6a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fc815ef70 con 0x7f3fc8162740 2024-04-24T23:48:32.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.748+0000 7f3fcff6a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f3fc80d09b0 con 0x7f3fc80b5500 2024-04-24T23:48:32.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.748+0000 7f3fcd505640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f3fc8162740 0x7f3fc80addb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.156:55474/0 (socket says 172.21.15.156:55474) 2024-04-24T23:48:32.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.748+0000 7f3fcdd06640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fc80b5500 0x7f3fc80d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.748+0000 7f3fcd505640 1 -- 172.21.15.156:0/3940052863 learned_addr learned my addr 172.21.15.156:0/3940052863 (peer_addr_for_me v1:172.21.15.156:0/0) 2024-04-24T23:48:32.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.748+0000 7f3fccd04640 1 -- 172.21.15.156:0/3940052863 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1123391418 0 0) 0x7f3fc815ef70 con 0x7f3fc8162740 2024-04-24T23:48:32.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.748+0000 7f3fcdd06640 1 -- 172.21.15.156:0/3940052863 >> v1:172.21.15.156:6789/0 conn(0x7f3fc8162740 legacy=0x7f3fc80addb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.749+0000 7f3fcdd06640 1 -- 172.21.15.156:0/3940052863 >> v1:172.21.15.16:6789/0 conn(0x7f3fc80ae4c0 legacy=0x7f3fc80cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.749+0000 7f3fcdd06640 1 -- 172.21.15.156:0/3940052863 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fc815f250 con 0x7f3fc80b5500 2024-04-24T23:48:32.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.749+0000 7f3fcdd06640 1 --2- 172.21.15.156:0/3940052863 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fc80b5500 0x7f3fc80d0470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40097d0 tx=0x7f3fc402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a084836635971ab server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:32.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.749+0000 7f3fccd04640 1 -- 172.21.15.156:0/3940052863 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4037070 con 0x7f3fc80b5500 2024-04-24T23:48:32.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.749+0000 7f3fccd04640 1 -- 172.21.15.156:0/3940052863 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fc4032440 con 0x7f3fc80b5500 2024-04-24T23:48:32.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.749+0000 7f3fccd04640 1 -- 172.21.15.156:0/3940052863 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4032810 con 0x7f3fc80b5500 2024-04-24T23:48:32.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.750+0000 7f3fcff6a640 1 -- 172.21.15.156:0/3940052863 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fc80b5500 msgr2=0x7f3fc80d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.750+0000 7f3fcff6a640 1 --2- 172.21.15.156:0/3940052863 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fc80b5500 0x7f3fc80d0470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40097d0 tx=0x7f3fc402a210 comp rx=0 tx=0).stop 2024-04-24T23:48:32.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.750+0000 7f3fcff6a640 1 -- 172.21.15.156:0/3940052863 shutdown_connections 2024-04-24T23:48:32.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.750+0000 7f3fcff6a640 1 --2- 172.21.15.156:0/3940052863 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fc80b5500 0x7f3fc80d0470 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.750+0000 7f3fcff6a640 1 -- 172.21.15.156:0/3940052863 >> 172.21.15.156:0/3940052863 conn(0x7f3fc80b2910 msgr2=0x7f3fc80b2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:32.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.750+0000 7f3fcff6a640 1 -- 172.21.15.156:0/3940052863 shutdown_connections 2024-04-24T23:48:32.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.750+0000 7f3fcff6a640 1 -- 172.21.15.156:0/3940052863 wait complete. 2024-04-24T23:48:32.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.751+0000 7f3fcff6a640 1 Processor -- start 2024-04-24T23:48:32.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.751+0000 7f3fcff6a640 1 -- start start 2024-04-24T23:48:32.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcff6a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fc80ae4c0 0x7f3fc8146370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcff6a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fc80b5500 0x7f3fc8158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcff6a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fc8162740 0x7f3fc8159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.753 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcff6a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc8162220 con 0x7f3fc8162740 2024-04-24T23:48:32.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcff6a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3fc8161f40 con 0x7f3fc80ae4c0 2024-04-24T23:48:32.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcff6a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f3fc80aea30 con 0x7f3fc80b5500 2024-04-24T23:48:32.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcdd06640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fc80ae4c0 0x7f3fc8146370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcdd06640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fc80ae4c0 0x7f3fc8146370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55048/0 (socket says 172.21.15.156:55048) 2024-04-24T23:48:32.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcdd06640 1 -- 172.21.15.156:0/1505213318 learned_addr learned my addr 172.21.15.156:0/1505213318 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-24T23:48:32.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcd505640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fc80b5500 0x7f3fc8158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fce507640 1 --2- 172.21.15.156:0/1505213318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fc8162740 0x7f3fc8159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcdd06640 1 -- 172.21.15.156:0/1505213318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fc80b5500 msgr2=0x7f3fc8158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcdd06640 1 --2- 172.21.15.156:0/1505213318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fc80b5500 0x7f3fc8158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcdd06640 1 -- 172.21.15.156:0/1505213318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fc8162740 msgr2=0x7f3fc8159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcdd06640 1 --2- 172.21.15.156:0/1505213318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fc8162740 0x7f3fc8159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcdd06640 1 -- 172.21.15.156:0/1505213318 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fc8146fe0 con 0x7f3fc80ae4c0 2024-04-24T23:48:32.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.752+0000 7f3fcd505640 1 --2- 172.21.15.156:0/1505213318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fc80b5500 0x7f3fc8158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:32.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.753+0000 7f3fcdd06640 1 --2- 172.21.15.156:0/1505213318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fc80ae4c0 0x7f3fc8146370 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40097d0 tx=0x7f3fc40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:32.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.753+0000 7f3fba7fc640 1 -- 172.21.15.156:0/1505213318 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4037040 con 0x7f3fc80ae4c0 2024-04-24T23:48:32.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.753+0000 7f3fba7fc640 1 -- 172.21.15.156:0/1505213318 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fc4042070 con 0x7f3fc80ae4c0 2024-04-24T23:48:32.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.753+0000 7f3fba7fc640 1 -- 172.21.15.156:0/1505213318 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fc4030bd0 con 0x7f3fc80ae4c0 2024-04-24T23:48:32.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.753+0000 7f3fcff6a640 1 -- 172.21.15.156:0/1505213318 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fc81472c0 con 0x7f3fc80ae4c0 2024-04-24T23:48:32.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.753+0000 7f3fcff6a640 1 -- 172.21.15.156:0/1505213318 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fc8147880 con 0x7f3fc80ae4c0 2024-04-24T23:48:32.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.753+0000 7f3fba7fc640 1 -- 172.21.15.156:0/1505213318 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3fc4047460 con 0x7f3fc80ae4c0 2024-04-24T23:48:32.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.753+0000 7f3fba7fc640 1 -- 172.21.15.156:0/1505213318 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f3fc403da30 con 0x7f3fc80ae4c0 2024-04-24T23:48:32.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.754+0000 7f3fcff6a640 1 -- 172.21.15.156:0/1505213318 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fc8068f90 con 0x7f3fc80ae4c0 2024-04-24T23:48:32.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.756+0000 7f3fba7fc640 1 -- 172.21.15.156:0/1505213318 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3fc4032e20 con 0x7f3fc80ae4c0 2024-04-24T23:48:32.769 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:32.770+0000 7fcfc9212040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T23:48:32.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.826+0000 7f3fcff6a640 1 -- 172.21.15.156:0/1505213318 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4f095fbc-1077-4a2d-bde6-98d4dd3b0931", "id": 6} v 0) -- 0x7f3fc80b5f60 con 0x7f3fc80ae4c0 2024-04-24T23:48:32.829 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.829+0000 7f3fba7fc640 1 -- 172.21.15.156:0/1505213318 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4f095fbc-1077-4a2d-bde6-98d4dd3b0931", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f3fc403d440 con 0x7f3fc80ae4c0 2024-04-24T23:48:32.829 INFO:teuthology.orchestra.run.smithi156.stdout:6 2024-04-24T23:48:32.830 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.830+0000 7f3fcff6a640 1 -- 172.21.15.156:0/1505213318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fc80ae4c0 msgr2=0x7f3fc8146370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.830 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.830+0000 7f3fcff6a640 1 --2- 172.21.15.156:0/1505213318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fc80ae4c0 0x7f3fc8146370 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3fc40097d0 tx=0x7f3fc40046c0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.830 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.830+0000 7f3fcff6a640 1 -- 172.21.15.156:0/1505213318 shutdown_connections 2024-04-24T23:48:32.830 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.830+0000 7f3fcff6a640 1 --2- 172.21.15.156:0/1505213318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fc8162740 0x7f3fc8159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.830 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.830+0000 7f3fcff6a640 1 --2- 172.21.15.156:0/1505213318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3fc80b5500 0x7f3fc8158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.830+0000 7f3fcff6a640 1 --2- 172.21.15.156:0/1505213318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3fc80ae4c0 0x7f3fc8146370 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.831+0000 7f3fcff6a640 1 -- 172.21.15.156:0/1505213318 >> 172.21.15.156:0/1505213318 conn(0x7f3fc80b2910 msgr2=0x7f3fc80afa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:32.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.831+0000 7f3fcff6a640 1 -- 172.21.15.156:0/1505213318 shutdown_connections 2024-04-24T23:48:32.831 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.831+0000 7f3fcff6a640 1 -- 172.21.15.156:0/1505213318 wait complete. 2024-04-24T23:48:32.840 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --cluster ceph osd new 3ba94c69-3667-4d5a-8f3e-9d30f3c784e0 7 2024-04-24T23:48:32.853 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:32.852+0000 7fac8b91c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:48:32.886 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:32.886+0000 7fcfc9212040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T23:48:32.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.980+0000 7f9864116640 1 Processor -- start 2024-04-24T23:48:32.979 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.980+0000 7f9864116640 1 -- start start 2024-04-24T23:48:32.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.981+0000 7f9864116640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f985c144620 0x7f985c0d0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.981+0000 7f9864116640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f985c0b2aa0 0x7f985c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.981+0000 7f9864116640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f985c0d0870 con 0x7f985c0d2030 2024-04-24T23:48:32.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.981+0000 7f9864116640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f985c0d2410 con 0x7f985c144620 2024-04-24T23:48:32.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.981+0000 7f9864116640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f985c0b1a00 con 0x7f985c0b2aa0 2024-04-24T23:48:32.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.981+0000 7f98616b1640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f985c0b2aa0 0x7f985c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.981+0000 7f98616b1640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f985c0b2aa0 0x7f985c0d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55052/0 (socket says 172.21.15.156:55052) 2024-04-24T23:48:32.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.981+0000 7f98616b1640 1 -- 172.21.15.156:0/1545806789 learned_addr learned my addr 172.21.15.156:0/1545806789 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-24T23:48:32.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.981+0000 7f9861eb2640 1 --2- 172.21.15.156:0/1545806789 >> v2:172.21.15.16:3300/0 conn(0x7f985c144620 0x7f985c0d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.981 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.981+0000 7f98616b1640 1 -- 172.21.15.156:0/1545806789 >> v2:172.21.15.16:3300/0 conn(0x7f985c144620 msgr2=0x7f985c0d0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:32.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.981+0000 7f98616b1640 1 --2- 172.21.15.156:0/1545806789 >> v2:172.21.15.16:3300/0 conn(0x7f985c144620 0x7f985c0d0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.981+0000 7f98616b1640 1 -- 172.21.15.156:0/1545806789 >> v1:172.21.15.156:6789/0 conn(0x7f985c0d2030 legacy=0x7f985c06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.982 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.981+0000 7f98616b1640 1 -- 172.21.15.156:0/1545806789 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985c0b1ce0 con 0x7f985c0b2aa0 2024-04-24T23:48:32.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.982+0000 7f98616b1640 1 --2- 172.21.15.156:0/1545806789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f985c0b2aa0 0x7f985c0d1af0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f985c147e70 tx=0x7f984c02a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=304dcb68c5943c33 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:32.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.982+0000 7f9860eb0640 1 -- 172.21.15.156:0/1545806789 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984c037070 con 0x7f985c0b2aa0 2024-04-24T23:48:32.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.982+0000 7f9860eb0640 1 -- 172.21.15.156:0/1545806789 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f984c032440 con 0x7f985c0b2aa0 2024-04-24T23:48:32.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.982+0000 7f9860eb0640 1 -- 172.21.15.156:0/1545806789 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f984c032810 con 0x7f985c0b2aa0 2024-04-24T23:48:32.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.983+0000 7f9864116640 1 -- 172.21.15.156:0/1545806789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f985c0b2aa0 msgr2=0x7f985c0d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.983+0000 7f9864116640 1 --2- 172.21.15.156:0/1545806789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f985c0b2aa0 0x7f985c0d1af0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f985c147e70 tx=0x7f984c02a270 comp rx=0 tx=0).stop 2024-04-24T23:48:32.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.983+0000 7f9864116640 1 -- 172.21.15.156:0/1545806789 shutdown_connections 2024-04-24T23:48:32.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.983+0000 7f9864116640 1 --2- 172.21.15.156:0/1545806789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f985c0b2aa0 0x7f985c0d1af0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.983+0000 7f9864116640 1 --2- 172.21.15.156:0/1545806789 >> v2:172.21.15.16:3300/0 conn(0x7f985c144620 0x7f985c0d0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.983+0000 7f9864116640 1 -- 172.21.15.156:0/1545806789 >> 172.21.15.156:0/1545806789 conn(0x7f985c06b0e0 msgr2=0x7f985c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:32.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.983+0000 7f9864116640 1 -- 172.21.15.156:0/1545806789 shutdown_connections 2024-04-24T23:48:32.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.983+0000 7f9864116640 1 -- 172.21.15.156:0/1545806789 wait complete. 2024-04-24T23:48:32.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.984+0000 7f9864116640 1 Processor -- start 2024-04-24T23:48:32.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.984+0000 7f9864116640 1 -- start start 2024-04-24T23:48:32.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9864116640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f985c0b2aa0 0x7f985c1579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9864116640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985c0d2030 0x7f985c0b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9864116640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f985c144620 0x7f985c144c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:32.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9864116640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f985c0b2580 con 0x7f985c0d2030 2024-04-24T23:48:32.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9864116640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f985c0b22a0 con 0x7f985c144620 2024-04-24T23:48:32.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9864116640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f985c144a00 con 0x7f985c0b2aa0 2024-04-24T23:48:32.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9861eb2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f985c0b2aa0 0x7f985c1579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f98626b3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f985c144620 0x7f985c144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.990 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f98626b3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f985c144620 0x7f985c144c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55054/0 (socket says 172.21.15.156:55054) 2024-04-24T23:48:32.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9861eb2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f985c0b2aa0 0x7f985c1579a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.156:46670/0 (socket says 172.21.15.156:46670) 2024-04-24T23:48:32.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f98626b3640 1 -- 172.21.15.156:0/1037173408 learned_addr learned my addr 172.21.15.156:0/1037173408 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-04-24T23:48:32.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f98616b1640 1 --2- 172.21.15.156:0/1037173408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985c0d2030 0x7f985c0b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:32.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9861eb2640 1 -- 172.21.15.156:0/1037173408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f985c144620 msgr2=0x7f985c144c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9861eb2640 1 --2- 172.21.15.156:0/1037173408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f985c144620 0x7f985c144c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9861eb2640 1 -- 172.21.15.156:0/1037173408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985c0d2030 msgr2=0x7f985c0b4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:32.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9861eb2640 1 --2- 172.21.15.156:0/1037173408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985c0d2030 0x7f985c0b4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:32.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9861eb2640 1 -- 172.21.15.156:0/1037173408 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f985c157ee0 con 0x7f985c0b2aa0 2024-04-24T23:48:32.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9861eb2640 1 --2- 172.21.15.156:0/1037173408 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f985c0b2aa0 0x7f985c1579a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f985800e870 tx=0x7f985800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:32.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f98527fc640 1 -- 172.21.15.156:0/1037173408 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858017070 con 0x7f985c0b2aa0 2024-04-24T23:48:32.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f98527fc640 1 -- 172.21.15.156:0/1037173408 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f985800ee60 con 0x7f985c0b2aa0 2024-04-24T23:48:32.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.985+0000 7f9864116640 1 -- 172.21.15.156:0/1037173408 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f985c0b5ab0 con 0x7f985c0b2aa0 2024-04-24T23:48:32.991 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.986+0000 7f98527fc640 1 -- 172.21.15.156:0/1037173408 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985800cb60 con 0x7f985c0b2aa0 2024-04-24T23:48:32.992 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.986+0000 7f9864116640 1 -- 172.21.15.156:0/1037173408 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f985c0b6070 con 0x7f985c0b2aa0 2024-04-24T23:48:32.992 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.986+0000 7f98527fc640 1 -- 172.21.15.156:0/1037173408 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f985801a070 con 0x7f985c0b2aa0 2024-04-24T23:48:32.992 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.986+0000 7f98527fc640 1 -- 172.21.15.156:0/1037173408 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f9858013ad0 con 0x7f985c0b2aa0 2024-04-24T23:48:32.992 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.986+0000 7f9864116640 1 -- 172.21.15.156:0/1037173408 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f985c068f90 con 0x7f985c0b2aa0 2024-04-24T23:48:32.992 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:32.988+0000 7f98527fc640 1 -- 172.21.15.156:0/1037173408 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f985c157ee0 con 0x7f985c0b2aa0 2024-04-24T23:48:33.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:33.059+0000 7f9864116640 1 -- 172.21.15.156:0/1037173408 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd new", "uuid": "3ba94c69-3667-4d5a-8f3e-9d30f3c784e0", "id": 7} v 0) -- 0x7f985c0cbb80 con 0x7f985c0b2aa0 2024-04-24T23:48:33.061 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:33.062+0000 7f98527fc640 1 -- 172.21.15.156:0/1037173408 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3ba94c69-3667-4d5a-8f3e-9d30f3c784e0", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f985800cda0 con 0x7f985c0b2aa0 2024-04-24T23:48:33.061 INFO:teuthology.orchestra.run.smithi156.stdout:7 2024-04-24T23:48:33.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:33.063+0000 7f9864116640 1 -- 172.21.15.156:0/1037173408 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f985c0b2aa0 msgr2=0x7f985c1579a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:33.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:33.063+0000 7f9864116640 1 --2- 172.21.15.156:0/1037173408 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f985c0b2aa0 0x7f985c1579a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f985800e870 tx=0x7f985800ed30 comp rx=0 tx=0).stop 2024-04-24T23:48:33.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:33.063+0000 7f9864116640 1 -- 172.21.15.156:0/1037173408 shutdown_connections 2024-04-24T23:48:33.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:33.063+0000 7f9864116640 1 --2- 172.21.15.156:0/1037173408 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f985c144620 0x7f985c144c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:33.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:33.063+0000 7f9864116640 1 --2- 172.21.15.156:0/1037173408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f985c0d2030 0x7f985c0b4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:33.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:33.063+0000 7f9864116640 1 --2- 172.21.15.156:0/1037173408 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f985c0b2aa0 0x7f985c1579a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:33.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:33.063+0000 7f9864116640 1 -- 172.21.15.156:0/1037173408 >> 172.21.15.156:0/1037173408 conn(0x7f985c06b0e0 msgr2=0x7f985c06b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:33.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:33.063+0000 7f9864116640 1 -- 172.21.15.156:0/1037173408 shutdown_connections 2024-04-24T23:48:33.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:48:33.063+0000 7f9864116640 1 -- 172.21.15.156:0/1037173408 wait complete. 2024-04-24T23:48:33.073 INFO:tasks.ceph.osd.0:Restarting daemon 2024-04-24T23:48:33.073 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-04-24T23:48:33.075 INFO:tasks.ceph.osd.0:Started 2024-04-24T23:48:33.075 INFO:tasks.ceph.osd.1:Restarting daemon 2024-04-24T23:48:33.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-04-24T23:48:33.078 INFO:tasks.ceph.osd.1:Started 2024-04-24T23:48:33.078 INFO:tasks.ceph.osd.2:Restarting daemon 2024-04-24T23:48:33.078 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-04-24T23:48:33.080 INFO:tasks.ceph.osd.2:Started 2024-04-24T23:48:33.080 INFO:tasks.ceph.osd.3:Restarting daemon 2024-04-24T23:48:33.080 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-04-24T23:48:33.082 INFO:tasks.ceph.osd.3:Started 2024-04-24T23:48:33.082 INFO:tasks.ceph.osd.4:Restarting daemon 2024-04-24T23:48:33.082 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-04-24T23:48:33.116 INFO:tasks.ceph.osd.4:Started 2024-04-24T23:48:33.116 INFO:tasks.ceph.osd.5:Restarting daemon 2024-04-24T23:48:33.116 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-04-24T23:48:33.118 INFO:tasks.ceph.osd.5:Started 2024-04-24T23:48:33.118 INFO:tasks.ceph.osd.6:Restarting daemon 2024-04-24T23:48:33.118 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-04-24T23:48:33.120 INFO:tasks.ceph.osd.6:Started 2024-04-24T23:48:33.121 INFO:tasks.ceph.osd.7:Restarting daemon 2024-04-24T23:48:33.121 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-04-24T23:48:33.123 INFO:tasks.ceph.osd.7:Started 2024-04-24T23:48:33.123 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:48:33.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.241+0000 7f2407da5640 1 Processor -- start 2024-04-24T23:48:33.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.241+0000 7f2407da5640 1 -- start start 2024-04-24T23:48:33.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.242+0000 7f2407da5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24001622d0 0x7f24000adc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:33.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.242+0000 7f2407da5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24001472f0 0x7f24000b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:33.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.242+0000 7f2407da5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2400147b90 con 0x7f24000af940 2024-04-24T23:48:33.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.242+0000 7f2407da5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2400162b70 con 0x7f24001472f0 2024-04-24T23:48:33.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.242+0000 7f2407da5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f24000b0100 con 0x7f24001622d0 2024-04-24T23:48:33.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.242+0000 7f2406342640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24001472f0 0x7f24000b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:33.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.242+0000 7f2406342640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24001472f0 0x7f24000b64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59822/0 (socket says 172.21.15.16:59822) 2024-04-24T23:48:33.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.242+0000 7f2406342640 1 -- 172.21.15.16:0/1509264359 learned_addr learned my addr 172.21.15.16:0/1509264359 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:33.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.242+0000 7f2405340640 1 --2- 172.21.15.16:0/1509264359 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24001622d0 0x7f24000adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:33.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.242+0000 7f2406342640 1 -- 172.21.15.16:0/1509264359 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24001622d0 msgr2=0x7f24000adc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:33.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.242+0000 7f2406342640 1 --2- 172.21.15.16:0/1509264359 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24001622d0 0x7f24000adc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:33.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.242+0000 7f2406342640 1 -- 172.21.15.16:0/1509264359 >> v1:172.21.15.156:6789/0 conn(0x7f24000af940 legacy=0x7f2400161bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:33.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.242+0000 7f2406342640 1 -- 172.21.15.16:0/1509264359 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24000afd20 con 0x7f24001472f0 2024-04-24T23:48:33.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.243+0000 7f2406342640 1 --2- 172.21.15.16:0/1509264359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24001472f0 0x7f24000b64f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009e50 tx=0x7f23f802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=670dd38a5d4075d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:33.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.243+0000 7f2404b3f640 1 -- 172.21.15.16:0/1509264359 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f8036070 con 0x7f24001472f0 2024-04-24T23:48:33.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.243+0000 7f2404b3f640 1 -- 172.21.15.16:0/1509264359 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23f8031600 con 0x7f24001472f0 2024-04-24T23:48:33.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.243+0000 7f2404b3f640 1 -- 172.21.15.16:0/1509264359 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23f80319d0 con 0x7f24001472f0 2024-04-24T23:48:33.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.244+0000 7f2407da5640 1 -- 172.21.15.16:0/1509264359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24001472f0 msgr2=0x7f24000b64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:33.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.244+0000 7f2407da5640 1 --2- 172.21.15.16:0/1509264359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24001472f0 0x7f24000b64f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009e50 tx=0x7f23f802a710 comp rx=0 tx=0).stop 2024-04-24T23:48:33.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.244+0000 7f2407da5640 1 -- 172.21.15.16:0/1509264359 shutdown_connections 2024-04-24T23:48:33.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.244+0000 7f2407da5640 1 --2- 172.21.15.16:0/1509264359 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24001622d0 0x7f24000adc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:33.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.244+0000 7f2407da5640 1 --2- 172.21.15.16:0/1509264359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24001472f0 0x7f24000b64f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:33.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.244+0000 7f2407da5640 1 -- 172.21.15.16:0/1509264359 >> 172.21.15.16:0/1509264359 conn(0x7f2400071b40 msgr2=0x7f2400071f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:33.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.244+0000 7f2407da5640 1 -- 172.21.15.16:0/1509264359 shutdown_connections 2024-04-24T23:48:33.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.244+0000 7f2407da5640 1 -- 172.21.15.16:0/1509264359 wait complete. 2024-04-24T23:48:33.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.245+0000 7f2407da5640 1 Processor -- start 2024-04-24T23:48:33.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.245+0000 7f2407da5640 1 -- start start 2024-04-24T23:48:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.246+0000 7f2407da5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24000af940 0x7f24001444b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.246+0000 7f2407da5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24001472f0 0x7f24000caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.246+0000 7f2407da5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24001622d0 0x7f24000cb470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.246+0000 7f2407da5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f24000b0680 con 0x7f24001472f0 2024-04-24T23:48:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.246+0000 7f2407da5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f24000b03a0 con 0x7f24001622d0 2024-04-24T23:48:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.246+0000 7f2407da5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f24000cf7f0 con 0x7f24000af940 2024-04-24T23:48:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.246+0000 7f2405340640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24001472f0 0x7f24000caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.246+0000 7f2405340640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24001472f0 0x7f24000caf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59838/0 (socket says 172.21.15.16:59838) 2024-04-24T23:48:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.246+0000 7f2405340640 1 -- 172.21.15.16:0/3610351202 learned_addr learned my addr 172.21.15.16:0/3610351202 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.246+0000 7f2405b41640 1 --2- 172.21.15.16:0/3610351202 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24000af940 0x7f24001444b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.246+0000 7f2406342640 1 --2- 172.21.15.16:0/3610351202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24001622d0 0x7f24000cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.246+0000 7f2405340640 1 -- 172.21.15.16:0/3610351202 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24000af940 msgr2=0x7f24001444b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:33.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.246+0000 7f2405340640 1 --2- 172.21.15.16:0/3610351202 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24000af940 0x7f24001444b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.246+0000 7f2405340640 1 -- 172.21.15.16:0/3610351202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24001622d0 msgr2=0x7f24000cb470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.247+0000 7f2405340640 1 --2- 172.21.15.16:0/3610351202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24001622d0 0x7f24000cb470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.247+0000 7f2405340640 1 -- 172.21.15.16:0/3610351202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24000c94f0 con 0x7f24001472f0 2024-04-24T23:48:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.247+0000 7f2406342640 1 --2- 172.21.15.16:0/3610351202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24001622d0 0x7f24000cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.247+0000 7f2405b41640 1 --2- 172.21.15.16:0/3610351202 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24000af940 0x7f24001444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.247+0000 7f2405340640 1 --2- 172.21.15.16:0/3610351202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24001472f0 0x7f24000caf30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00e870 tx=0x7f23fc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.247+0000 7f23f67fc640 1 -- 172.21.15.16:0/3610351202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc017070 con 0x7f24001472f0 2024-04-24T23:48:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.247+0000 7f23f67fc640 1 -- 172.21.15.16:0/3610351202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23fc00c940 con 0x7f24001472f0 2024-04-24T23:48:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.247+0000 7f23f67fc640 1 -- 172.21.15.16:0/3610351202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23fc00cd30 con 0x7f24001472f0 2024-04-24T23:48:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.247+0000 7f2407da5640 1 -- 172.21.15.16:0/3610351202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24000c97d0 con 0x7f24001472f0 2024-04-24T23:48:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.247+0000 7f2407da5640 1 -- 172.21.15.16:0/3610351202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24000c9d90 con 0x7f24001472f0 2024-04-24T23:48:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.247+0000 7f23f67fc640 1 -- 172.21.15.16:0/3610351202 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f23fc01a070 con 0x7f24001472f0 2024-04-24T23:48:33.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.247+0000 7f23f67fc640 1 -- 172.21.15.16:0/3610351202 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f23fc01cb20 con 0x7f24001472f0 2024-04-24T23:48:33.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.248+0000 7f2407da5640 1 -- 172.21.15.16:0/3610351202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2400074a20 con 0x7f24001472f0 2024-04-24T23:48:33.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.250+0000 7f23f67fc640 1 -- 172.21.15.16:0/3610351202 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f23fc01c450 con 0x7f24001472f0 2024-04-24T23:48:33.283 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:33.283+0000 7fac8b91c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T23:48:33.300 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:33.300+0000 7fcfc9212040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T23:48:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.313+0000 7f2407da5640 1 -- 172.21.15.16:0/3610351202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f240004f5e0 con 0x7f24001472f0 2024-04-24T23:48:33.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.313+0000 7f23f67fc640 1 -- 172.21.15.16:0/3610351202 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f23fc01c630 con 0x7f24001472f0 2024-04-24T23:48:33.314 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:48:33.314 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":10,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:33.060932+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:48:33.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.314+0000 7f2407da5640 1 -- 172.21.15.16:0/3610351202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24001472f0 msgr2=0x7f24000caf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:33.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.314+0000 7f2407da5640 1 --2- 172.21.15.16:0/3610351202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24001472f0 0x7f24000caf30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00e870 tx=0x7f23fc00ed30 comp rx=0 tx=0).stop 2024-04-24T23:48:33.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.314+0000 7f2407da5640 1 -- 172.21.15.16:0/3610351202 shutdown_connections 2024-04-24T23:48:33.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.314+0000 7f2407da5640 1 --2- 172.21.15.16:0/3610351202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24001622d0 0x7f24000cb470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.314+0000 7f2407da5640 1 --2- 172.21.15.16:0/3610351202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24001472f0 0x7f24000caf30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.314+0000 7f2407da5640 1 --2- 172.21.15.16:0/3610351202 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24000af940 0x7f24001444b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.314+0000 7f2407da5640 1 -- 172.21.15.16:0/3610351202 >> 172.21.15.16:0/3610351202 conn(0x7f2400071b40 msgr2=0x7f24000b21c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.314+0000 7f2407da5640 1 -- 172.21.15.16:0/3610351202 shutdown_connections 2024-04-24T23:48:33.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:33.314+0000 7f2407da5640 1 -- 172.21.15.16:0/3610351202 wait complete. 2024-04-24T23:48:33.325 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-04-24T23:48:33.326 INFO:tasks.ceph:Waiting for OSDs to come up 2024-04-24T23:48:33.554 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:33.554+0000 7fac8b91c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T23:48:33.556 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:33.557+0000 7fcfc9212040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T23:48:33.608 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:33.609+0000 7fcfc9212040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T23:48:33.609 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:33.608+0000 7fac8b91c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T23:48:33.687 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-04-24T23:48:33.686+0000 7f222c5fc640 -1 Falling back to public interface 2024-04-24T23:48:33.687 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-04-24T23:48:33.686+0000 7faee932c640 -1 Falling back to public interface 2024-04-24T23:48:33.688 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-04-24T23:48:33.687+0000 7faee82db640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2024-04-24T23:48:33.696 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-04-24T23:48:33.695+0000 7f93c8a20640 -1 Falling back to public interface 2024-04-24T23:48:33.699 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-04-24T23:48:33.698+0000 7feedb42f640 -1 Falling back to public interface 2024-04-24T23:48:33.717 INFO:tasks.ceph.osd.5.smithi156.stderr:2024-04-24T23:48:33.717+0000 7f1390d21640 -1 Falling back to public interface 2024-04-24T23:48:33.717 INFO:tasks.ceph.osd.4.smithi156.stderr:2024-04-24T23:48:33.718+0000 7f489380c640 -1 Falling back to public interface 2024-04-24T23:48:33.724 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:33.725+0000 7fcfc9212040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T23:48:33.733 INFO:tasks.ceph.osd.6.smithi156.stderr:2024-04-24T23:48:33.733+0000 7f393f4d7640 -1 Falling back to public interface 2024-04-24T23:48:33.735 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:33.734+0000 7fac8b91c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T23:48:33.738 INFO:tasks.ceph.osd.7.smithi156.stderr:2024-04-24T23:48:33.738+0000 7fdd3e7cb640 -1 Falling back to public interface 2024-04-24T23:48:34.008 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:34.008+0000 7fcfc9212040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T23:48:34.028 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:34.027+0000 7fac8b91c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T23:48:34.056 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:34.057+0000 7fcfc9212040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T23:48:34.077 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:34.076+0000 7fac8b91c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T23:48:34.132 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T23:48:34.210 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:34.211+0000 7fcfc9212040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T23:48:34.228 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.224+0000 7f1c6671b640 1 Processor -- start 2024-04-24T23:48:34.228 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.224+0000 7f1c6671b640 1 -- start start 2024-04-24T23:48:34.229 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c6671b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1c6015e1a0 0x7f1c600d2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:34.229 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c6671b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c600b3130 0x7f1c600cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:34.229 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c6671b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c600ce9d0 con 0x7f1c600d0680 2024-04-24T23:48:34.229 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c6671b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f1c6015e580 con 0x7f1c6015e1a0 2024-04-24T23:48:34.229 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c6671b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1c600cec40 con 0x7f1c600b3130 2024-04-24T23:48:34.229 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c5f7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c600b3130 0x7f1c600cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:34.229 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c5ffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1c6015e1a0 0x7f1c600d2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:34.229 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c5f7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c600b3130 0x7f1c600cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:46768/0 (socket says 172.21.15.16:46768) 2024-04-24T23:48:34.229 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c5ffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1c6015e1a0 0x7f1c600d2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:43614/0 (socket says 172.21.15.16:43614) 2024-04-24T23:48:34.229 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c5f7fe640 1 -- 172.21.15.16:0/3469970485 learned_addr learned my addr 172.21.15.16:0/3469970485 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:34.229 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c5f7fe640 1 -- 172.21.15.16:0/3469970485 >> v2:172.21.15.156:3300/0 conn(0x7f1c6015e1a0 msgr2=0x7f1c600d2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:34.229 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c5f7fe640 1 --2- 172.21.15.16:0/3469970485 >> v2:172.21.15.156:3300/0 conn(0x7f1c6015e1a0 0x7f1c600d2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:34.229 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c5f7fe640 1 -- 172.21.15.16:0/3469970485 >> v1:172.21.15.156:6789/0 conn(0x7f1c600d0680 legacy=0x7f1c600b3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:34.229 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c5f7fe640 1 -- 172.21.15.16:0/3469970485 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c600cef20 con 0x7f1c600b3130 2024-04-24T23:48:34.230 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.225+0000 7f1c5ffff640 1 --2- 172.21.15.16:0/3469970485 >> v2:172.21.15.156:3300/0 conn(0x7f1c6015e1a0 0x7f1c600d2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:34.230 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.226+0000 7f1c5f7fe640 1 --2- 172.21.15.16:0/3469970485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c600b3130 0x7f1c600cf720 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009870 tx=0x7f1c5002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4babe85dd79a9964 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:34.230 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.226+0000 7f1c5effd640 1 -- 172.21.15.16:0/3469970485 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c50037070 con 0x7f1c600b3130 2024-04-24T23:48:34.230 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.226+0000 7f1c5effd640 1 -- 172.21.15.16:0/3469970485 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c50032440 con 0x7f1c600b3130 2024-04-24T23:48:34.230 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.226+0000 7f1c5effd640 1 -- 172.21.15.16:0/3469970485 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c50032810 con 0x7f1c600b3130 2024-04-24T23:48:34.230 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.227+0000 7f1c6671b640 1 -- 172.21.15.16:0/3469970485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c600b3130 msgr2=0x7f1c600cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:34.230 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.227+0000 7f1c6671b640 1 --2- 172.21.15.16:0/3469970485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c600b3130 0x7f1c600cf720 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009870 tx=0x7f1c5002a270 comp rx=0 tx=0).stop 2024-04-24T23:48:34.230 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.227+0000 7f1c6671b640 1 -- 172.21.15.16:0/3469970485 shutdown_connections 2024-04-24T23:48:34.230 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.227+0000 7f1c6671b640 1 --2- 172.21.15.16:0/3469970485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c600b3130 0x7f1c600cf720 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:34.230 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.227+0000 7f1c6671b640 1 --2- 172.21.15.16:0/3469970485 >> v2:172.21.15.156:3300/0 conn(0x7f1c6015e1a0 0x7f1c600d2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:34.230 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.227+0000 7f1c6671b640 1 -- 172.21.15.16:0/3469970485 >> 172.21.15.16:0/3469970485 conn(0x7f1c600d3d90 msgr2=0x7f1c600d41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:34.230 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.227+0000 7f1c6671b640 1 -- 172.21.15.16:0/3469970485 shutdown_connections 2024-04-24T23:48:34.230 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.227+0000 7f1c6671b640 1 -- 172.21.15.16:0/3469970485 wait complete. 2024-04-24T23:48:34.233 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.228+0000 7f1c6671b640 1 Processor -- start 2024-04-24T23:48:34.233 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.228+0000 7f1c6671b640 1 -- start start 2024-04-24T23:48:34.233 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.229+0000 7f1c6671b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c600b3130 0x7f1c6015f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:34.233 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.229+0000 7f1c6671b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c600d0680 0x7f1c6015c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:34.233 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.229+0000 7f1c6671b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c6015e1a0 0x7f1c600b1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:34.233 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.229+0000 7f1c6671b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1c600cfc60 con 0x7f1c6015e1a0 2024-04-24T23:48:34.233 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.229+0000 7f1c6671b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1c600d28b0 con 0x7f1c600b3130 2024-04-24T23:48:34.233 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.229+0000 7f1c6671b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1c600d2ba0 con 0x7f1c600d0680 2024-04-24T23:48:34.233 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.229+0000 7f1c64cb8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c6015e1a0 0x7f1c600b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:34.233 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.229+0000 7f1c5f7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c600d0680 0x7f1c6015c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:34.234 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.229+0000 7f1c5ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c600b3130 0x7f1c6015f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:34.234 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.229+0000 7f1c64cb8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c6015e1a0 0x7f1c600b1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59844/0 (socket says 172.21.15.16:59844) 2024-04-24T23:48:34.234 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.229+0000 7f1c64cb8640 1 -- 172.21.15.16:0/2798728309 learned_addr learned my addr 172.21.15.16:0/2798728309 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:34.234 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.229+0000 7f1c5ffff640 1 -- 172.21.15.16:0/2798728309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c600d0680 msgr2=0x7f1c6015c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:34.234 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.229+0000 7f1c5ffff640 1 --2- 172.21.15.16:0/2798728309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c600d0680 0x7f1c6015c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:34.234 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.230+0000 7f1c5ffff640 1 -- 172.21.15.16:0/2798728309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c6015e1a0 msgr2=0x7f1c600b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:34.234 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.230+0000 7f1c5ffff640 1 --2- 172.21.15.16:0/2798728309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c6015e1a0 0x7f1c600b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:34.234 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.230+0000 7f1c5ffff640 1 -- 172.21.15.16:0/2798728309 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c60160030 con 0x7f1c600b3130 2024-04-24T23:48:34.234 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.230+0000 7f1c5f7fe640 1 --2- 172.21.15.16:0/2798728309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c600d0680 0x7f1c6015c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:34.234 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.230+0000 7f1c64cb8640 1 --2- 172.21.15.16:0/2798728309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c6015e1a0 0x7f1c600b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:34.234 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.230+0000 7f1c5ffff640 1 --2- 172.21.15.16:0/2798728309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c600b3130 0x7f1c6015f8c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1c5400e900 tx=0x7f1c5400edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:34.234 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.230+0000 7f1c5cff9640 1 -- 172.21.15.16:0/2798728309 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c54017070 con 0x7f1c600b3130 2024-04-24T23:48:34.234 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.230+0000 7f1c5cff9640 1 -- 172.21.15.16:0/2798728309 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c5400ca00 con 0x7f1c600b3130 2024-04-24T23:48:34.235 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.230+0000 7f1c5cff9640 1 -- 172.21.15.16:0/2798728309 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5400cdf0 con 0x7f1c600b3130 2024-04-24T23:48:34.235 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.230+0000 7f1c6671b640 1 -- 172.21.15.16:0/2798728309 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c60160310 con 0x7f1c600b3130 2024-04-24T23:48:34.235 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.230+0000 7f1c6671b640 1 -- 172.21.15.16:0/2798728309 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c600b70a0 con 0x7f1c600b3130 2024-04-24T23:48:34.235 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.230+0000 7f1c5cff9640 1 -- 172.21.15.16:0/2798728309 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1c5401a070 con 0x7f1c600b3130 2024-04-24T23:48:34.235 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.230+0000 7f1c5cff9640 1 -- 172.21.15.16:0/2798728309 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f1c5401cb20 con 0x7f1c600b3130 2024-04-24T23:48:34.235 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.231+0000 7f1c6671b640 1 -- 172.21.15.16:0/2798728309 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c6004f570 con 0x7f1c600b3130 2024-04-24T23:48:34.235 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.233+0000 7f1c5cff9640 1 -- 172.21.15.16:0/2798728309 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1c540092e0 con 0x7f1c600b3130 2024-04-24T23:48:34.243 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:34.242+0000 7fac8b91c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T23:48:34.296 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.295+0000 7f1c6671b640 1 -- 172.21.15.16:0/2798728309 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1c600b6e70 con 0x7f1c600b3130 2024-04-24T23:48:34.296 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.296+0000 7f1c5cff9640 1 -- 172.21.15.16:0/2798728309 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f1c5401c450 con 0x7f1c600b3130 2024-04-24T23:48:34.297 INFO:teuthology.misc.health.smithi016.stdout: 2024-04-24T23:48:34.297 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":10,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:33.060932+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:48:34.298 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.297+0000 7f1c6671b640 1 -- 172.21.15.16:0/2798728309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c600b3130 msgr2=0x7f1c6015f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:34.298 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.297+0000 7f1c6671b640 1 --2- 172.21.15.16:0/2798728309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c600b3130 0x7f1c6015f8c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1c5400e900 tx=0x7f1c5400edc0 comp rx=0 tx=0).stop 2024-04-24T23:48:34.298 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.297+0000 7f1c6671b640 1 -- 172.21.15.16:0/2798728309 shutdown_connections 2024-04-24T23:48:34.298 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.297+0000 7f1c6671b640 1 --2- 172.21.15.16:0/2798728309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c6015e1a0 0x7f1c600b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:34.298 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.297+0000 7f1c6671b640 1 --2- 172.21.15.16:0/2798728309 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1c600d0680 0x7f1c6015c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:34.298 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.297+0000 7f1c6671b640 1 --2- 172.21.15.16:0/2798728309 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1c600b3130 0x7f1c6015f8c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:34.298 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.297+0000 7f1c6671b640 1 -- 172.21.15.16:0/2798728309 >> 172.21.15.16:0/2798728309 conn(0x7f1c600d3d90 msgr2=0x7f1c600d4170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:34.298 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.297+0000 7f1c6671b640 1 -- 172.21.15.16:0/2798728309 shutdown_connections 2024-04-24T23:48:34.298 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:34.297+0000 7f1c6671b640 1 -- 172.21.15.16:0/2798728309 wait complete. 2024-04-24T23:48:34.307 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-24T23:48:34.348 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:34.349+0000 7fcfc9212040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T23:48:34.388 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:34.388+0000 7fac8b91c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T23:48:34.397 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:34.398+0000 7fcfc9212040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T23:48:34.440 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:34.439+0000 7fac8b91c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T23:48:34.466 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:34.467+0000 7fcfc9212040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T23:48:34.511 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:34.511+0000 7fcfc9212040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T23:48:34.513 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:34.512+0000 7fac8b91c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T23:48:34.560 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:34.560+0000 7fac8b91c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T23:48:34.990 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:34.991+0000 7fcfc9212040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T23:48:35.059 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:35.058+0000 7fac8b91c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T23:48:35.106 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:35.106+0000 7fcfc9212040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T23:48:35.184 INFO:tasks.ceph.mgr.y.smithi016.stderr:2024-04-24T23:48:35.183+0000 7fac8b91c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T23:48:36.121 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:36.122+0000 7fcf698fc640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-24T23:48:36.122 INFO:tasks.ceph.mgr.x.smithi156.stderr:2024-04-24T23:48:36.122+0000 7fcf698fc640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-24T23:48:37.145 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-04-24T23:48:37.145+0000 7faee932c640 -1 osd.2 0 log_to_monitors true 2024-04-24T23:48:37.181 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-04-24T23:48:37.181+0000 7f222c5fc640 -1 osd.1 0 log_to_monitors true 2024-04-24T23:48:37.200 INFO:tasks.ceph.osd.5.smithi156.stderr:2024-04-24T23:48:37.200+0000 7f1390d21640 -1 osd.5 0 log_to_monitors true 2024-04-24T23:48:37.218 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-04-24T23:48:37.218+0000 7f93c8a20640 -1 osd.3 0 log_to_monitors true 2024-04-24T23:48:37.218 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-04-24T23:48:37.218+0000 7feedb42f640 -1 osd.0 0 log_to_monitors true 2024-04-24T23:48:37.244 INFO:tasks.ceph.osd.6.smithi156.stderr:2024-04-24T23:48:37.244+0000 7f393f4d7640 -1 osd.6 0 log_to_monitors true 2024-04-24T23:48:37.254 INFO:tasks.ceph.osd.4.smithi156.stderr:2024-04-24T23:48:37.255+0000 7f489380c640 -1 osd.4 0 log_to_monitors true 2024-04-24T23:48:37.268 INFO:tasks.ceph.osd.7.smithi156.stderr:2024-04-24T23:48:37.268+0000 7fdd3e7cb640 -1 osd.7 0 log_to_monitors true 2024-04-24T23:48:40.435 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-04-24T23:48:40.433+0000 7feed6980640 -1 osd.0 0 waiting for initial osdmap 2024-04-24T23:48:40.449 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-04-24T23:48:40.448+0000 7feece7b1640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:48:40.503 INFO:tasks.ceph.osd.7.smithi156.stderr:2024-04-24T23:48:40.504+0000 7fdd3a761640 -1 osd.7 0 waiting for initial osdmap 2024-04-24T23:48:40.522 INFO:tasks.ceph.osd.7.smithi156.stderr:2024-04-24T23:48:40.523+0000 7fdd32581640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:48:40.534 INFO:tasks.ceph.osd.6.smithi156.stderr:2024-04-24T23:48:40.535+0000 7f393b46d640 -1 osd.6 0 waiting for initial osdmap 2024-04-24T23:48:40.547 INFO:tasks.ceph.osd.6.smithi156.stderr:2024-04-24T23:48:40.548+0000 7f393328d640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:48:40.552 INFO:tasks.ceph.osd.4.smithi156.stderr:2024-04-24T23:48:40.553+0000 7f488f1ce640 -1 osd.4 0 waiting for initial osdmap 2024-04-24T23:48:40.562 INFO:tasks.ceph.osd.4.smithi156.stderr:2024-04-24T23:48:40.563+0000 7f4886fee640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:48:40.575 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-04-24T23:48:40.575+0000 7f93c4317640 -1 osd.3 0 waiting for initial osdmap 2024-04-24T23:48:40.589 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-04-24T23:48:40.590+0000 7faee52c2640 -1 osd.2 0 waiting for initial osdmap 2024-04-24T23:48:40.592 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-04-24T23:48:40.593+0000 7f93bc137640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:48:40.598 INFO:tasks.ceph.osd.5.smithi156.stderr:2024-04-24T23:48:40.598+0000 7f138ccb7640 -1 osd.5 0 waiting for initial osdmap 2024-04-24T23:48:40.604 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-04-24T23:48:40.605+0000 7faedd0e2640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:48:40.611 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-04-24T23:48:40.609+0000 7f2228da6640 -1 osd.1 0 waiting for initial osdmap 2024-04-24T23:48:40.616 INFO:tasks.ceph.osd.5.smithi156.stderr:2024-04-24T23:48:40.617+0000 7f1384ad7640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:48:40.619 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-04-24T23:48:40.620+0000 7f22203b2640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T23:48:41.113 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T23:48:41.213 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.213+0000 7f15d09e2640 1 Processor -- start 2024-04-24T23:48:41.213 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.213+0000 7f15d09e2640 1 -- start start 2024-04-24T23:48:41.214 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.214+0000 7f15d09e2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f15cc15dda0 0x7f15cc0d0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:41.215 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.214+0000 7f15d09e2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15cc0cec60 0x7f15cc0d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:41.215 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.214+0000 7f15d09e2640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15cc06c4c0 con 0x7f15cc15e180 2024-04-24T23:48:41.215 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.214+0000 7f15d09e2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f15cc06c270 con 0x7f15cc15dda0 2024-04-24T23:48:41.215 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.214+0000 7f15d09e2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f15cc06c7a0 con 0x7f15cc0cec60 2024-04-24T23:48:41.215 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.214+0000 7f15c9d9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f15cc15e180 0x7f15cc0ce550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57574/0 (socket says 172.21.15.16:57574) 2024-04-24T23:48:41.215 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.214+0000 7f15c9d9b640 1 -- 172.21.15.16:0/3262563571 learned_addr learned my addr 172.21.15.16:0/3262563571 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:48:41.215 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.214+0000 7f15ca59c640 1 --2- 172.21.15.16:0/3262563571 >> v2:172.21.15.156:3300/0 conn(0x7f15cc15dda0 0x7f15cc0d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:41.215 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.214+0000 7f15cad9d640 1 --2- 172.21.15.16:0/3262563571 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15cc0cec60 0x7f15cc0d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:41.215 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.215+0000 7f15c959a640 1 -- 172.21.15.16:0/3262563571 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 120670382 0 0) 0x7f15cc06c4c0 con 0x7f15cc15e180 2024-04-24T23:48:41.215 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.215+0000 7f15c959a640 1 -- 172.21.15.16:0/3262563571 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15b4003520 con 0x7f15cc15e180 2024-04-24T23:48:41.215 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.215+0000 7f15ca59c640 1 -- 172.21.15.16:0/3262563571 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15cc0cec60 msgr2=0x7f15cc0d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:41.215 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.215+0000 7f15ca59c640 1 --2- 172.21.15.16:0/3262563571 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15cc0cec60 0x7f15cc0d0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:41.215 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.215+0000 7f15ca59c640 1 -- 172.21.15.16:0/3262563571 >> v1:172.21.15.16:6789/0 conn(0x7f15cc15e180 legacy=0x7f15cc0ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:41.216 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.215+0000 7f15ca59c640 1 -- 172.21.15.16:0/3262563571 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15cc06ca80 con 0x7f15cc15dda0 2024-04-24T23:48:41.216 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.215+0000 7f15cad9d640 1 --2- 172.21.15.16:0/3262563571 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15cc0cec60 0x7f15cc0d0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:41.217 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.216+0000 7f15ca59c640 1 --2- 172.21.15.16:0/3262563571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc15dda0 0x7f15cc0d0890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f15bc0097a0 tx=0x7f15bc02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76d5e1754c340e9b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:41.217 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.216+0000 7f15c959a640 1 -- 172.21.15.16:0/3262563571 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc037070 con 0x7f15cc15dda0 2024-04-24T23:48:41.217 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.216+0000 7f15c959a640 1 -- 172.21.15.16:0/3262563571 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15bc032440 con 0x7f15cc15dda0 2024-04-24T23:48:41.217 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.216+0000 7f15c959a640 1 -- 172.21.15.16:0/3262563571 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15bc032810 con 0x7f15cc15dda0 2024-04-24T23:48:41.217 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.216+0000 7f15d09e2640 1 -- 172.21.15.16:0/3262563571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc15dda0 msgr2=0x7f15cc0d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:41.217 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.216+0000 7f15d09e2640 1 --2- 172.21.15.16:0/3262563571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc15dda0 0x7f15cc0d0890 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f15bc0097a0 tx=0x7f15bc02a180 comp rx=0 tx=0).stop 2024-04-24T23:48:41.218 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.216+0000 7f15d09e2640 1 -- 172.21.15.16:0/3262563571 shutdown_connections 2024-04-24T23:48:41.218 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.216+0000 7f15d09e2640 1 --2- 172.21.15.16:0/3262563571 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15cc0cec60 0x7f15cc0d0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:41.218 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.216+0000 7f15d09e2640 1 --2- 172.21.15.16:0/3262563571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc15dda0 0x7f15cc0d0890 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:41.218 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.216+0000 7f15d09e2640 1 -- 172.21.15.16:0/3262563571 >> 172.21.15.16:0/3262563571 conn(0x7f15cc0b3680 msgr2=0x7f15cc0b3a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:41.218 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.216+0000 7f15d09e2640 1 -- 172.21.15.16:0/3262563571 shutdown_connections 2024-04-24T23:48:41.218 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.216+0000 7f15d09e2640 1 -- 172.21.15.16:0/3262563571 wait complete. 2024-04-24T23:48:41.218 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.218+0000 7f15d09e2640 1 Processor -- start 2024-04-24T23:48:41.218 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.218+0000 7f15d09e2640 1 -- start start 2024-04-24T23:48:41.221 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.218+0000 7f15d09e2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc0cec60 0x7f15cc0c9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:41.221 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.218+0000 7f15d09e2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc15dda0 0x7f15cc0b6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:41.221 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.218+0000 7f15d09e2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15cc0ca920 0x7f15cc0cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:41.221 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.218+0000 7f15d09e2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc06d560 con 0x7f15cc15dda0 2024-04-24T23:48:41.221 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.218+0000 7f15d09e2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f15cc06d040 con 0x7f15cc0cec60 2024-04-24T23:48:41.221 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.218+0000 7f15d09e2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f15cc159530 con 0x7f15cc0ca920 2024-04-24T23:48:41.221 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15c9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc15dda0 0x7f15cc0b6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:41.221 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15c9d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc15dda0 0x7f15cc0b6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60434/0 (socket says 172.21.15.16:60434) 2024-04-24T23:48:41.221 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15c9d9b640 1 -- 172.21.15.16:0/3728079980 learned_addr learned my addr 172.21.15.16:0/3728079980 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:41.221 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15ca59c640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc0cec60 0x7f15cc0c9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:41.221 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15cad9d640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15cc0ca920 0x7f15cc0cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:41.221 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15c9d9b640 1 -- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15cc0ca920 msgr2=0x7f15cc0cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:41.222 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15c9d9b640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15cc0ca920 0x7f15cc0cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:41.222 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15c9d9b640 1 -- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc0cec60 msgr2=0x7f15cc0c9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:41.222 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15c9d9b640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc0cec60 0x7f15cc0c9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:41.222 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15c9d9b640 1 -- 172.21.15.16:0/3728079980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15cc0c8ce0 con 0x7f15cc15dda0 2024-04-24T23:48:41.222 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15cad9d640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15cc0ca920 0x7f15cc0cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:41.222 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15ca59c640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc0cec60 0x7f15cc0c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:41.222 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15c9d9b640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc15dda0 0x7f15cc0b6ef0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f15c000c460 tx=0x7f15c000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:41.222 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15baffd640 1 -- 172.21.15.16:0/3728079980 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c0017070 con 0x7f15cc15dda0 2024-04-24T23:48:41.222 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15baffd640 1 -- 172.21.15.16:0/3728079980 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15c0005ce0 con 0x7f15cc15dda0 2024-04-24T23:48:41.222 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15baffd640 1 -- 172.21.15.16:0/3728079980 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15c0007290 con 0x7f15cc15dda0 2024-04-24T23:48:41.222 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15d09e2640 1 -- 172.21.15.16:0/3728079980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15cc0c8fc0 con 0x7f15cc15dda0 2024-04-24T23:48:41.222 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.219+0000 7f15d09e2640 1 -- 172.21.15.16:0/3728079980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15cc0c9580 con 0x7f15cc15dda0 2024-04-24T23:48:41.222 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.220+0000 7f15d09e2640 1 -- 172.21.15.16:0/3728079980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15cc0c9780 con 0x7f15cc15dda0 2024-04-24T23:48:41.223 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.221+0000 7f15baffd640 1 -- 172.21.15.16:0/3728079980 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f15c001a070 con 0x7f15cc15dda0 2024-04-24T23:48:41.223 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.221+0000 7f15baffd640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f15ac06bda0 0x7f15ac06e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:41.223 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.221+0000 7f15baffd640 1 -- 172.21.15.16:0/3728079980 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7f15c0089310 con 0x7f15cc15dda0 2024-04-24T23:48:41.223 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.221+0000 7f15ca59c640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f15ac06bda0 0x7f15ac06e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:41.223 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.222+0000 7f15ca59c640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f15ac06bda0 0x7f15ac06e260 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f15bc034000 tx=0x7f15bc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:41.225 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.226+0000 7f15baffd640 1 -- 172.21.15.16:0/3728079980 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f15cc0c9780 con 0x7f15cc15dda0 2024-04-24T23:48:41.408 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.409+0000 7f15d09e2640 1 -- 172.21.15.16:0/3728079980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15cc0cb550 con 0x7f15cc15dda0 2024-04-24T23:48:41.409 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.409+0000 7f15baffd640 1 -- 172.21.15.16:0/3728079980 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7f15c0058b30 con 0x7f15cc15dda0 2024-04-24T23:48:41.409 INFO:teuthology.misc.health.smithi016.stdout: 2024-04-24T23:48:41.409 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":12,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:39.124722+0000","last_up_change":"0.000000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:48:41.411 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.411+0000 7f15d09e2640 1 -- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f15ac06bda0 msgr2=0x7f15ac06e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:41.411 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.411+0000 7f15d09e2640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f15ac06bda0 0x7f15ac06e260 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f15bc034000 tx=0x7f15bc02e000 comp rx=0 tx=0).stop 2024-04-24T23:48:41.411 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.411+0000 7f15d09e2640 1 -- 172.21.15.16:0/3728079980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc15dda0 msgr2=0x7f15cc0b6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:41.411 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.411+0000 7f15d09e2640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc15dda0 0x7f15cc0b6ef0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f15c000c460 tx=0x7f15c000c920 comp rx=0 tx=0).stop 2024-04-24T23:48:41.411 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.411+0000 7f15d09e2640 1 -- 172.21.15.16:0/3728079980 shutdown_connections 2024-04-24T23:48:41.411 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.411+0000 7f15d09e2640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f15ac06bda0 0x7f15ac06e260 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:41.412 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.411+0000 7f15d09e2640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15cc0ca920 0x7f15cc0cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:41.412 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.411+0000 7f15d09e2640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15cc15dda0 0x7f15cc0b6ef0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:41.412 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.411+0000 7f15d09e2640 1 --2- 172.21.15.16:0/3728079980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15cc0cec60 0x7f15cc0c9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:41.412 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.411+0000 7f15d09e2640 1 -- 172.21.15.16:0/3728079980 >> 172.21.15.16:0/3728079980 conn(0x7f15cc0b3680 msgr2=0x7f15cc06fd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:41.412 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.411+0000 7f15d09e2640 1 -- 172.21.15.16:0/3728079980 shutdown_connections 2024-04-24T23:48:41.412 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:41.411+0000 7f15d09e2640 1 -- 172.21.15.16:0/3728079980 wait complete. 2024-04-24T23:48:41.420 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-04-24T23:48:48.228 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-04-24T23:48:48.332 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.333+0000 7f08795ee640 1 Processor -- start 2024-04-24T23:48:48.333 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.333+0000 7f08795ee640 1 -- start start 2024-04-24T23:48:48.335 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.333+0000 7f08795ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08740d1960 0x7f08740b4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.335 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.333+0000 7f08795ee640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f08740d2a60 0x7f08740b4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.335 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.333+0000 7f08795ee640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08740b35b0 con 0x7f087415de80 2024-04-24T23:48:48.335 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.333+0000 7f08795ee640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f08740b3bc0 con 0x7f08740d2a60 2024-04-24T23:48:48.335 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.333+0000 7f08795ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f08740d1df0 con 0x7f08740d1960 2024-04-24T23:48:48.335 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.334+0000 7f0872ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f087415de80 0x7f08740d2350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60704/0 (socket says 172.21.15.16:60704) 2024-04-24T23:48:48.335 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.334+0000 7f0872ffd640 1 -- 172.21.15.16:0/1173950405 learned_addr learned my addr 172.21.15.16:0/1173950405 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:48:48.335 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.334+0000 7f08727fc640 1 --2- 172.21.15.16:0/1173950405 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08740d1960 0x7f08740b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.335 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.334+0000 7f08737fe640 1 --2- 172.21.15.16:0/1173950405 >> v2:172.21.15.156:3300/0 conn(0x7f08740d2a60 0x7f08740b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.335 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.334+0000 7f0871ffb640 1 -- 172.21.15.16:0/1173950405 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 130498338 0 0) 0x7f08740b35b0 con 0x7f087415de80 2024-04-24T23:48:48.336 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.334+0000 7f0871ffb640 1 -- 172.21.15.16:0/1173950405 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0860003520 con 0x7f087415de80 2024-04-24T23:48:48.336 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.334+0000 7f08727fc640 1 -- 172.21.15.16:0/1173950405 >> v2:172.21.15.156:3300/0 conn(0x7f08740d2a60 msgr2=0x7f08740b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:48.336 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.334+0000 7f08727fc640 1 --2- 172.21.15.16:0/1173950405 >> v2:172.21.15.156:3300/0 conn(0x7f08740d2a60 0x7f08740b4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.336 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.334+0000 7f08727fc640 1 -- 172.21.15.16:0/1173950405 >> v1:172.21.15.16:6789/0 conn(0x7f087415de80 legacy=0x7f08740d2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.336 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.334+0000 7f08727fc640 1 -- 172.21.15.16:0/1173950405 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08740b3850 con 0x7f08740d1960 2024-04-24T23:48:48.336 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.334+0000 7f08737fe640 1 --2- 172.21.15.16:0/1173950405 >> v2:172.21.15.156:3300/0 conn(0x7f08740d2a60 0x7f08740b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:48.336 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.335+0000 7f08727fc640 1 --2- 172.21.15.16:0/1173950405 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08740d1960 0x7f08740b4050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f085c0097a0 tx=0x7f085c02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=572d77ad7764026 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:48.336 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.335+0000 7f0871ffb640 1 -- 172.21.15.16:0/1173950405 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c037070 con 0x7f08740d1960 2024-04-24T23:48:48.336 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.335+0000 7f0871ffb640 1 -- 172.21.15.16:0/1173950405 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085c032440 con 0x7f08740d1960 2024-04-24T23:48:48.336 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.335+0000 7f0871ffb640 1 -- 172.21.15.16:0/1173950405 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c032810 con 0x7f08740d1960 2024-04-24T23:48:48.339 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.336+0000 7f08795ee640 1 -- 172.21.15.16:0/1173950405 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08740d1960 msgr2=0x7f08740b4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.339 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.336+0000 7f08795ee640 1 --2- 172.21.15.16:0/1173950405 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08740d1960 0x7f08740b4050 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f085c0097a0 tx=0x7f085c02a210 comp rx=0 tx=0).stop 2024-04-24T23:48:48.339 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.336+0000 7f08795ee640 1 -- 172.21.15.16:0/1173950405 shutdown_connections 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.336+0000 7f08795ee640 1 --2- 172.21.15.16:0/1173950405 >> v2:172.21.15.156:3300/0 conn(0x7f08740d2a60 0x7f08740b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.336+0000 7f08795ee640 1 --2- 172.21.15.16:0/1173950405 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08740d1960 0x7f08740b4050 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.336+0000 7f08795ee640 1 -- 172.21.15.16:0/1173950405 >> 172.21.15.16:0/1173950405 conn(0x7f087409d060 msgr2=0x7f08740d3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.336+0000 7f08795ee640 1 -- 172.21.15.16:0/1173950405 shutdown_connections 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.336+0000 7f08795ee640 1 -- 172.21.15.16:0/1173950405 wait complete. 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.337+0000 7f08795ee640 1 Processor -- start 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.337+0000 7f08795ee640 1 -- start start 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08795ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08740d2a60 0x7f087414ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08795ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087415de80 0x7f087414d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08795ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08740cdad0 0x7f087415f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08795ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f087415e3e0 con 0x7f087415de80 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08795ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f087415e260 con 0x7f08740d2a60 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08795ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f08740d47d0 con 0x7f08740cdad0 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08727fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087415de80 0x7f087414d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.340 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08727fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087415de80 0x7f087414d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43602/0 (socket says 172.21.15.16:43602) 2024-04-24T23:48:48.341 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08727fc640 1 -- 172.21.15.16:0/1598154719 learned_addr learned my addr 172.21.15.16:0/1598154719 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:48.341 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08737fe640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08740cdad0 0x7f087415f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.341 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f0872ffd640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08740d2a60 0x7f087414ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.341 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08727fc640 1 -- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08740cdad0 msgr2=0x7f087415f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.341 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08727fc640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08740cdad0 0x7f087415f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.341 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08727fc640 1 -- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08740d2a60 msgr2=0x7f087414ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.341 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08727fc640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08740d2a60 0x7f087414ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.341 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08727fc640 1 -- 172.21.15.16:0/1598154719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08741604a0 con 0x7f087415de80 2024-04-24T23:48:48.341 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08737fe640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08740cdad0 0x7f087415f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:48.341 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.338+0000 7f08727fc640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087415de80 0x7f087414d360 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f085c035040 tx=0x7f085c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:48.341 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.339+0000 7f086b7fe640 1 -- 172.21.15.16:0/1598154719 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c041030 con 0x7f087415de80 2024-04-24T23:48:48.341 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.339+0000 7f086b7fe640 1 -- 172.21.15.16:0/1598154719 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f085c03d4e0 con 0x7f087415de80 2024-04-24T23:48:48.341 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.339+0000 7f08795ee640 1 -- 172.21.15.16:0/1598154719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0874160780 con 0x7f087415de80 2024-04-24T23:48:48.341 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.339+0000 7f086b7fe640 1 -- 172.21.15.16:0/1598154719 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c037040 con 0x7f087415de80 2024-04-24T23:48:48.342 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.339+0000 7f08795ee640 1 -- 172.21.15.16:0/1598154719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0874160d40 con 0x7f087415de80 2024-04-24T23:48:48.342 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.340+0000 7f08795ee640 1 -- 172.21.15.16:0/1598154719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f087414aab0 con 0x7f087415de80 2024-04-24T23:48:48.342 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.340+0000 7f086b7fe640 1 -- 172.21.15.16:0/1598154719 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f085c044070 con 0x7f087415de80 2024-04-24T23:48:48.343 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.341+0000 7f086b7fe640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f085406be30 0x7f085406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.343 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.341+0000 7f086b7fe640 1 -- 172.21.15.16:0/1598154719 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f085c0ab220 con 0x7f087415de80 2024-04-24T23:48:48.343 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.341+0000 7f0872ffd640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f085406be30 0x7f085406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.343 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.341+0000 7f0872ffd640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f085406be30 0x7f085406e2f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f086c007fd0 tx=0x7f086c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:48.345 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.345+0000 7f086b7fe640 1 -- 172.21.15.16:0/1598154719 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f087414aab0 con 0x7f087415de80 2024-04-24T23:48:48.531 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.532+0000 7f08795ee640 1 -- 172.21.15.16:0/1598154719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f087404f570 con 0x7f087415de80 2024-04-24T23:48:48.532 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.532+0000 7f086b7fe640 1 -- 172.21.15.16:0/1598154719 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12872 (secure 0 0 0) 0x7f085c044e40 con 0x7f087415de80 2024-04-24T23:48:48.532 INFO:teuthology.misc.health.smithi016.stdout: 2024-04-24T23:48:48.532 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":17,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:45.450145+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:48:48.535 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.534+0000 7f08795ee640 1 -- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f085406be30 msgr2=0x7f085406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.535 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.534+0000 7f08795ee640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f085406be30 0x7f085406e2f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f086c007fd0 tx=0x7f086c009040 comp rx=0 tx=0).stop 2024-04-24T23:48:48.535 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.535+0000 7f08795ee640 1 -- 172.21.15.16:0/1598154719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087415de80 msgr2=0x7f087414d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.535 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.535+0000 7f08795ee640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087415de80 0x7f087414d360 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f085c035040 tx=0x7f085c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.535 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.535+0000 7f08795ee640 1 -- 172.21.15.16:0/1598154719 shutdown_connections 2024-04-24T23:48:48.535 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.535+0000 7f08795ee640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f085406be30 0x7f085406e2f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.535 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.535+0000 7f08795ee640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f08740cdad0 0x7f087415f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.535 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.535+0000 7f08795ee640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f087415de80 0x7f087414d360 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.535 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.535+0000 7f08795ee640 1 --2- 172.21.15.16:0/1598154719 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08740d2a60 0x7f087414ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.535 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.535+0000 7f08795ee640 1 -- 172.21.15.16:0/1598154719 >> 172.21.15.16:0/1598154719 conn(0x7f087409d060 msgr2=0x7f08740b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:48.535 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.535+0000 7f08795ee640 1 -- 172.21.15.16:0/1598154719 shutdown_connections 2024-04-24T23:48:48.535 INFO:teuthology.misc.health.smithi016.stderr:2024-04-24T23:48:48.535+0000 7f08795ee640 1 -- 172.21.15.16:0/1598154719 wait complete. 2024-04-24T23:48:48.544 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-04-24T23:48:48.544 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-04-24T23:48:48.545 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-24T23:48:48.545 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-24T23:48:48.547 INFO:tasks.ceph.mds.a:Started 2024-04-24T23:48:48.547 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-24T23:48:48.547 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-24T23:48:48.549 INFO:tasks.ceph.mds.c:Started 2024-04-24T23:48:48.549 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-24T23:48:48.550 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-24T23:48:48.555 INFO:tasks.ceph.mds.b:Started 2024-04-24T23:48:48.555 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-24T23:48:48.555 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-24T23:48:48.560 INFO:tasks.ceph.mds.d:Started 2024-04-24T23:48:48.560 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-04-24T23:48:48.560 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:48:48.622 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-04-24T23:48:48.622 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-04-24T23:48:48.639 INFO:tasks.ceph.mds.d.smithi156.stdout:starting mds.d at 2024-04-24T23:48:48.639 INFO:tasks.ceph.mds.b.smithi156.stdout:starting mds.b at 2024-04-24T23:48:48.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.664+0000 7f0dbca9a640 1 Processor -- start 2024-04-24T23:48:48.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.665+0000 7f0dbca9a640 1 -- start start 2024-04-24T23:48:48.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.665+0000 7f0dbca9a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0db815e0c0 0x7f0db80ceba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.665+0000 7f0dbca9a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0db8162730 0x7f0db80b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.665+0000 7f0dbca9a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0db80d20e0 con 0x7f0db80b75f0 2024-04-24T23:48:48.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.665+0000 7f0dbca9a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f0db80d23c0 con 0x7f0db8162730 2024-04-24T23:48:48.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.665+0000 7f0dbca9a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0db80d26a0 con 0x7f0db815e0c0 2024-04-24T23:48:48.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.666+0000 7f0db659c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0db815e0c0 0x7f0db80ceba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.666+0000 7f0db5d9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0db8162730 0x7f0db80b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.666+0000 7f0db6d9d640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f0db80b75f0 0x7f0db80d3180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:50986/0 (socket says 172.21.15.16:50986) 2024-04-24T23:48:48.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.666+0000 7f0db659c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0db815e0c0 0x7f0db80ceba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:41492/0 (socket says 172.21.15.16:41492) 2024-04-24T23:48:48.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.666+0000 7f0db6d9d640 1 -- 172.21.15.16:0/2929056598 learned_addr learned my addr 172.21.15.16:0/2929056598 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:48:48.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.666+0000 7f0db5d9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0db8162730 0x7f0db80b70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:52060/0 (socket says 172.21.15.16:52060) 2024-04-24T23:48:48.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.666+0000 7f0db659c640 1 -- 172.21.15.16:0/2929056598 >> v2:172.21.15.156:3300/0 conn(0x7f0db8162730 msgr2=0x7f0db80b70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:48.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.666+0000 7f0db659c640 1 --2- 172.21.15.16:0/2929056598 >> v2:172.21.15.156:3300/0 conn(0x7f0db8162730 0x7f0db80b70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.666+0000 7f0db559a640 1 -- 172.21.15.16:0/2929056598 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 333093752 0 0) 0x7f0db80d20e0 con 0x7f0db80b75f0 2024-04-24T23:48:48.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.666+0000 7f0db659c640 1 -- 172.21.15.16:0/2929056598 >> v1:172.21.15.156:6789/0 conn(0x7f0db80b75f0 legacy=0x7f0db80d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.666+0000 7f0db659c640 1 -- 172.21.15.16:0/2929056598 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db80d2980 con 0x7f0db815e0c0 2024-04-24T23:48:48.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.667+0000 7f0db659c640 1 --2- 172.21.15.16:0/2929056598 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0db815e0c0 0x7f0db80ceba0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0db00097a0 tx=0x7f0db002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8778ca99f8cc39d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:48.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.667+0000 7f0db559a640 1 -- 172.21.15.16:0/2929056598 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0037070 con 0x7f0db815e0c0 2024-04-24T23:48:48.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.667+0000 7f0db559a640 1 -- 172.21.15.16:0/2929056598 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db0032440 con 0x7f0db815e0c0 2024-04-24T23:48:48.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.667+0000 7f0db559a640 1 -- 172.21.15.16:0/2929056598 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db0032810 con 0x7f0db815e0c0 2024-04-24T23:48:48.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.667+0000 7f0dbca9a640 1 -- 172.21.15.16:0/2929056598 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0db815e0c0 msgr2=0x7f0db80ceba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.667+0000 7f0dbca9a640 1 --2- 172.21.15.16:0/2929056598 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0db815e0c0 0x7f0db80ceba0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0db00097a0 tx=0x7f0db002a180 comp rx=0 tx=0).stop 2024-04-24T23:48:48.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.668+0000 7f0dbca9a640 1 -- 172.21.15.16:0/2929056598 shutdown_connections 2024-04-24T23:48:48.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.668+0000 7f0dbca9a640 1 --2- 172.21.15.16:0/2929056598 >> v2:172.21.15.156:3300/0 conn(0x7f0db8162730 0x7f0db80b70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.668+0000 7f0dbca9a640 1 --2- 172.21.15.16:0/2929056598 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0db815e0c0 0x7f0db80ceba0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.668+0000 7f0dbca9a640 1 -- 172.21.15.16:0/2929056598 >> 172.21.15.16:0/2929056598 conn(0x7f0db80d41f0 msgr2=0x7f0db80d4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:48.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.668+0000 7f0dbca9a640 1 -- 172.21.15.16:0/2929056598 shutdown_connections 2024-04-24T23:48:48.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.668+0000 7f0dbca9a640 1 -- 172.21.15.16:0/2929056598 wait complete. 2024-04-24T23:48:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.669+0000 7f0dbca9a640 1 Processor -- start 2024-04-24T23:48:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.669+0000 7f0dbca9a640 1 -- start start 2024-04-24T23:48:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0dbca9a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0db80b75f0 0x7f0db815ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0dbca9a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0db815e0c0 0x7f0db80c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0dbca9a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db8162730 0x7f0db80cc850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0dbca9a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0db815e4a0 con 0x7f0db8162730 2024-04-24T23:48:48.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0dbca9a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0db8162b90 con 0x7f0db815e0c0 2024-04-24T23:48:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0dbca9a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0db815e620 con 0x7f0db80b75f0 2024-04-24T23:48:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0db6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db8162730 0x7f0db80cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0db6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db8162730 0x7f0db80cc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43656/0 (socket says 172.21.15.16:43656) 2024-04-24T23:48:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0db6d9d640 1 -- 172.21.15.16:0/793694421 learned_addr learned my addr 172.21.15.16:0/793694421 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0db5d9b640 1 --2- 172.21.15.16:0/793694421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0db815e0c0 0x7f0db80c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0db659c640 1 --2- 172.21.15.16:0/793694421 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0db80b75f0 0x7f0db815ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0db6d9d640 1 -- 172.21.15.16:0/793694421 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0db80b75f0 msgr2=0x7f0db815ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0db6d9d640 1 --2- 172.21.15.16:0/793694421 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0db80b75f0 0x7f0db815ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0db6d9d640 1 -- 172.21.15.16:0/793694421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0db815e0c0 msgr2=0x7f0db80c99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0db6d9d640 1 --2- 172.21.15.16:0/793694421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0db815e0c0 0x7f0db80c99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0db6d9d640 1 -- 172.21.15.16:0/793694421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0db8160e30 con 0x7f0db8162730 2024-04-24T23:48:48.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0db6d9d640 1 --2- 172.21.15.16:0/793694421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db8162730 0x7f0db80cc850 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0da800c7a0 tx=0x7f0da800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0daeffd640 1 -- 172.21.15.16:0/793694421 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da8018070 con 0x7f0db8162730 2024-04-24T23:48:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0daeffd640 1 -- 172.21.15.16:0/793694421 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0da800a640 con 0x7f0db8162730 2024-04-24T23:48:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0dbca9a640 1 -- 172.21.15.16:0/793694421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0db8161110 con 0x7f0db8162730 2024-04-24T23:48:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.670+0000 7f0daeffd640 1 -- 172.21.15.16:0/793694421 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0da800aab0 con 0x7f0db8162730 2024-04-24T23:48:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.671+0000 7f0dbca9a640 1 -- 172.21.15.16:0/793694421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0db815a950 con 0x7f0db8162730 2024-04-24T23:48:48.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.671+0000 7f0dacff9640 1 -- 172.21.15.16:0/793694421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d90005290 con 0x7f0db8162730 2024-04-24T23:48:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.672+0000 7f0daeffd640 1 -- 172.21.15.16:0/793694421 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0da800ac50 con 0x7f0db8162730 2024-04-24T23:48:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.672+0000 7f0daeffd640 1 --2- 172.21.15.16:0/793694421 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d9806be30 0x7f0d9806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.673+0000 7f0daeffd640 1 -- 172.21.15.16:0/793694421 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f0da808a160 con 0x7f0db8162730 2024-04-24T23:48:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.673+0000 7f0db659c640 1 --2- 172.21.15.16:0/793694421 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d9806be30 0x7f0d9806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.673+0000 7f0db659c640 1 --2- 172.21.15.16:0/793694421 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d9806be30 0x7f0d9806e2f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0db0034000 tx=0x7f0db002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:48.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.677+0000 7f0daeffd640 1 -- 172.21.15.16:0/793694421 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0da8058790 con 0x7f0db8162730 2024-04-24T23:48:48.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.860+0000 7f0dacff9640 1 -- 172.21.15.16:0/793694421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d90005850 con 0x7f0db8162730 2024-04-24T23:48:48.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.861+0000 7f0daeffd640 1 -- 172.21.15.16:0/793694421 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12872 (secure 0 0 0) 0x7f0da805ca90 con 0x7f0db8162730 2024-04-24T23:48:48.861 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:48:48.861 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:45.450145+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:48:48.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.862+0000 7f0dacff9640 1 -- 172.21.15.16:0/793694421 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d9806be30 msgr2=0x7f0d9806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.862+0000 7f0dacff9640 1 --2- 172.21.15.16:0/793694421 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d9806be30 0x7f0d9806e2f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0db0034000 tx=0x7f0db002e000 comp rx=0 tx=0).stop 2024-04-24T23:48:48.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.862+0000 7f0dacff9640 1 -- 172.21.15.16:0/793694421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db8162730 msgr2=0x7f0db80cc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.862+0000 7f0dacff9640 1 --2- 172.21.15.16:0/793694421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db8162730 0x7f0db80cc850 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0da800c7a0 tx=0x7f0da800cc60 comp rx=0 tx=0).stop 2024-04-24T23:48:48.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.863+0000 7f0dacff9640 1 -- 172.21.15.16:0/793694421 shutdown_connections 2024-04-24T23:48:48.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.863+0000 7f0dacff9640 1 --2- 172.21.15.16:0/793694421 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d9806be30 0x7f0d9806e2f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.863+0000 7f0dacff9640 1 --2- 172.21.15.16:0/793694421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0db8162730 0x7f0db80cc850 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.863+0000 7f0dacff9640 1 --2- 172.21.15.16:0/793694421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0db815e0c0 0x7f0db80c99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.863+0000 7f0dacff9640 1 --2- 172.21.15.16:0/793694421 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0db80b75f0 0x7f0db815ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.863+0000 7f0dacff9640 1 -- 172.21.15.16:0/793694421 >> 172.21.15.16:0/793694421 conn(0x7f0db80d41f0 msgr2=0x7f0db80b3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:48.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.863+0000 7f0dacff9640 1 -- 172.21.15.16:0/793694421 shutdown_connections 2024-04-24T23:48:48.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.863+0000 7f0dacff9640 1 -- 172.21.15.16:0/793694421 wait complete. 2024-04-24T23:48:48.873 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T23:48:48.874 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:48:48.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.977+0000 7f1bc05b7640 1 Processor -- start 2024-04-24T23:48:48.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.978+0000 7f1bc05b7640 1 -- start start 2024-04-24T23:48:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.978+0000 7f1bc05b7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1bb80d3050 0x7f1bb80b7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.978+0000 7f1bc05b7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb815e450 con 0x7f1bb80b6f90 2024-04-24T23:48:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.978+0000 7f1bc05b7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bb806ca40 con 0x7f1bb815e070 2024-04-24T23:48:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.978+0000 7f1bc05b7640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1bb80cf3a0 con 0x7f1bb80d3050 2024-04-24T23:48:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.979+0000 7f1bbdb52640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1bb80b6f90 0x7f1bb80d2940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60716/0 (socket says 172.21.15.16:60716) 2024-04-24T23:48:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.979+0000 7f1bbdb52640 1 -- 172.21.15.16:0/3460234728 learned_addr learned my addr 172.21.15.16:0/3460234728 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:48:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.979+0000 7f1bbeb54640 1 --2- 172.21.15.16:0/3460234728 >> v2:172.21.15.16:3300/0 conn(0x7f1bb80d3050 0x7f1bb80b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.979+0000 7f1bbeb54640 1 -- 172.21.15.16:0/3460234728 >> v1:172.21.15.156:6789/0 conn(0x7f1bb815e070 legacy=0x7f1bb806c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.979+0000 7f1bbeb54640 1 -- 172.21.15.16:0/3460234728 >> v1:172.21.15.16:6789/0 conn(0x7f1bb80b6f90 legacy=0x7f1bb80d2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.979+0000 7f1bbeb54640 1 -- 172.21.15.16:0/3460234728 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb80ce790 con 0x7f1bb80d3050 2024-04-24T23:48:48.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.979+0000 7f1bbd351640 1 -- 172.21.15.16:0/3460234728 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3227508283 0 0) 0x7f1bb815e450 con 0x7f1bb80b6f90 2024-04-24T23:48:48.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.980+0000 7f1bbeb54640 1 --2- 172.21.15.16:0/3460234728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb80d3050 0x7f1bb80b7410 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0009d90 tx=0x7f1bb002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba4a1eaec72337dd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:48.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.980+0000 7f1bbd351640 1 -- 172.21.15.16:0/3460234728 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0007740 con 0x7f1bb80d3050 2024-04-24T23:48:48.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.980+0000 7f1bbd351640 1 -- 172.21.15.16:0/3460234728 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1bb00078e0 con 0x7f1bb80d3050 2024-04-24T23:48:48.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.980+0000 7f1bbd351640 1 -- 172.21.15.16:0/3460234728 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb0007cb0 con 0x7f1bb80d3050 2024-04-24T23:48:48.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.980+0000 7f1bc05b7640 1 -- 172.21.15.16:0/3460234728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb80d3050 msgr2=0x7f1bb80b7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.980+0000 7f1bc05b7640 1 --2- 172.21.15.16:0/3460234728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb80d3050 0x7f1bb80b7410 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1bb0009d90 tx=0x7f1bb002a660 comp rx=0 tx=0).stop 2024-04-24T23:48:48.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.980+0000 7f1bc05b7640 1 -- 172.21.15.16:0/3460234728 shutdown_connections 2024-04-24T23:48:48.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.980+0000 7f1bc05b7640 1 --2- 172.21.15.16:0/3460234728 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb80d3050 0x7f1bb80b7410 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.980+0000 7f1bc05b7640 1 -- 172.21.15.16:0/3460234728 >> 172.21.15.16:0/3460234728 conn(0x7f1bb80b5850 msgr2=0x7f1bb80b5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:48.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.980+0000 7f1bc05b7640 1 -- 172.21.15.16:0/3460234728 shutdown_connections 2024-04-24T23:48:48.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.980+0000 7f1bc05b7640 1 -- 172.21.15.16:0/3460234728 wait complete. 2024-04-24T23:48:48.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.982+0000 7f1bc05b7640 1 Processor -- start 2024-04-24T23:48:48.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.982+0000 7f1bc05b7640 1 -- start start 2024-04-24T23:48:48.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.982+0000 7f1bc05b7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1bb80b6f90 0x7f1bb81f4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.982+0000 7f1bc05b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb80d3050 0x7f1bb8158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.982+0000 7f1bc05b7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bb815e070 0x7f1bb81f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.982+0000 7f1bc05b7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb80cf030 con 0x7f1bb80d3050 2024-04-24T23:48:48.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.982+0000 7f1bc05b7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb80ced50 con 0x7f1bb815e070 2024-04-24T23:48:48.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.982+0000 7f1bc05b7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1bb80ce3b0 con 0x7f1bb80b6f90 2024-04-24T23:48:48.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.982+0000 7f1bbdb52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb80d3050 0x7f1bb8158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.982+0000 7f1bbdb52640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb80d3050 0x7f1bb8158bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43680/0 (socket says 172.21.15.16:43680) 2024-04-24T23:48:48.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.982+0000 7f1bbdb52640 1 -- 172.21.15.16:0/2091153386 learned_addr learned my addr 172.21.15.16:0/2091153386 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:48.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.983+0000 7f1bbeb54640 1 --2- 172.21.15.16:0/2091153386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bb815e070 0x7f1bb81f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.983+0000 7f1bbe353640 1 --2- 172.21.15.16:0/2091153386 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1bb80b6f90 0x7f1bb81f4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.983+0000 7f1bbdb52640 1 -- 172.21.15.16:0/2091153386 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1bb80b6f90 msgr2=0x7f1bb81f4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.983+0000 7f1bbdb52640 1 --2- 172.21.15.16:0/2091153386 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1bb80b6f90 0x7f1bb81f4740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.983+0000 7f1bbdb52640 1 -- 172.21.15.16:0/2091153386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bb815e070 msgr2=0x7f1bb81f72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:48.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.983+0000 7f1bbdb52640 1 --2- 172.21.15.16:0/2091153386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bb815e070 0x7f1bb81f72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:48.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.983+0000 7f1bbdb52640 1 -- 172.21.15.16:0/2091153386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb81f3cc0 con 0x7f1bb80d3050 2024-04-24T23:48:48.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.983+0000 7f1bbdb52640 1 --2- 172.21.15.16:0/2091153386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb80d3050 0x7f1bb8158bd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800c3f0 tx=0x7f1ba800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:48.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.983+0000 7f1baeffd640 1 -- 172.21.15.16:0/2091153386 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba80074e0 con 0x7f1bb80d3050 2024-04-24T23:48:48.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.983+0000 7f1baeffd640 1 -- 172.21.15.16:0/2091153386 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ba80047e0 con 0x7f1bb80d3050 2024-04-24T23:48:48.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.983+0000 7f1bc05b7640 1 -- 172.21.15.16:0/2091153386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb81f7d70 con 0x7f1bb80d3050 2024-04-24T23:48:48.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.983+0000 7f1baeffd640 1 -- 172.21.15.16:0/2091153386 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba8004c10 con 0x7f1bb80d3050 2024-04-24T23:48:48.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.983+0000 7f1bc05b7640 1 -- 172.21.15.16:0/2091153386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb81f3f40 con 0x7f1bb80d3050 2024-04-24T23:48:48.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.984+0000 7f1bc05b7640 1 -- 172.21.15.16:0/2091153386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bb8068f90 con 0x7f1bb80d3050 2024-04-24T23:48:48.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.985+0000 7f1baeffd640 1 -- 172.21.15.16:0/2091153386 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1ba8011070 con 0x7f1bb80d3050 2024-04-24T23:48:48.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.985+0000 7f1baeffd640 1 --2- 172.21.15.16:0/2091153386 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1b9c06be60 0x7f1b9c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:48.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.985+0000 7f1baeffd640 1 -- 172.21.15.16:0/2091153386 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f1ba808a560 con 0x7f1bb80d3050 2024-04-24T23:48:48.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.985+0000 7f1bbe353640 1 --2- 172.21.15.16:0/2091153386 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1b9c06be60 0x7f1b9c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:48.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.986+0000 7f1bbe353640 1 --2- 172.21.15.16:0/2091153386 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1b9c06be60 0x7f1b9c06e320 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4000f60 tx=0x7f1bb4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:48.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:48.989+0000 7f1baeffd640 1 -- 172.21.15.16:0/2091153386 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1bb81f3cc0 con 0x7f1bb80d3050 2024-04-24T23:48:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.170+0000 7f1bc05b7640 1 -- 172.21.15.16:0/2091153386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1bb815c6d0 con 0x7f1bb80d3050 2024-04-24T23:48:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.171+0000 7f1baeffd640 1 -- 172.21.15.16:0/2091153386 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f1ba8058c10 con 0x7f1bb80d3050 2024-04-24T23:48:49.171 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.172+0000 7f1bc05b7640 1 -- 172.21.15.16:0/2091153386 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1b9c06be60 msgr2=0x7f1b9c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.173+0000 7f1bc05b7640 1 --2- 172.21.15.16:0/2091153386 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1b9c06be60 0x7f1b9c06e320 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1bb4000f60 tx=0x7f1bb4009040 comp rx=0 tx=0).stop 2024-04-24T23:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.173+0000 7f1bc05b7640 1 -- 172.21.15.16:0/2091153386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb80d3050 msgr2=0x7f1bb8158bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.173+0000 7f1bc05b7640 1 --2- 172.21.15.16:0/2091153386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb80d3050 0x7f1bb8158bd0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1ba800c3f0 tx=0x7f1ba800c8b0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.173+0000 7f1bc05b7640 1 -- 172.21.15.16:0/2091153386 shutdown_connections 2024-04-24T23:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.173+0000 7f1bc05b7640 1 --2- 172.21.15.16:0/2091153386 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1b9c06be60 0x7f1b9c06e320 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.173+0000 7f1bc05b7640 1 --2- 172.21.15.16:0/2091153386 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1bb815e070 0x7f1bb81f72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.173+0000 7f1bc05b7640 1 --2- 172.21.15.16:0/2091153386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1bb80d3050 0x7f1bb8158bd0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.173+0000 7f1bc05b7640 1 --2- 172.21.15.16:0/2091153386 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1bb80b6f90 0x7f1bb81f4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.173+0000 7f1bc05b7640 1 -- 172.21.15.16:0/2091153386 >> 172.21.15.16:0/2091153386 conn(0x7f1bb80b5850 msgr2=0x7f1bb8161910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.173+0000 7f1bc05b7640 1 -- 172.21.15.16:0/2091153386 shutdown_connections 2024-04-24T23:48:49.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.173+0000 7f1bc05b7640 1 -- 172.21.15.16:0/2091153386 wait complete. 2024-04-24T23:48:49.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:48:49.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.284+0000 7efefb7f9640 1 Processor -- start 2024-04-24T23:48:49.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.284+0000 7efefb7f9640 1 -- start start 2024-04-24T23:48:49.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.285+0000 7efefb7f9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7efef415a450 0x7efef40b2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:49.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.285+0000 7efefb7f9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efef40b3300 con 0x7efef40729a0 2024-04-24T23:48:49.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.285+0000 7efefb7f9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efef40b35e0 con 0x7efef406e910 2024-04-24T23:48:49.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.285+0000 7efefb7f9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7efef40b38c0 con 0x7efef415a450 2024-04-24T23:48:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.285+0000 7efef9d96640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efef40729a0 0x7efef40ccc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60732/0 (socket says 172.21.15.16:60732) 2024-04-24T23:48:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.285+0000 7efef9d96640 1 -- 172.21.15.16:0/790970314 learned_addr learned my addr 172.21.15.16:0/790970314 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:48:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.285+0000 7efef9595640 1 --2- 172.21.15.16:0/790970314 >> v2:172.21.15.156:3300/0 conn(0x7efef415a450 0x7efef40b2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.285+0000 7efef3fff640 1 -- 172.21.15.16:0/790970314 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2354797503 0 0) 0x7efef40b3300 con 0x7efef40729a0 2024-04-24T23:48:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.285+0000 7efef3fff640 1 -- 172.21.15.16:0/790970314 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efedc003520 con 0x7efef40729a0 2024-04-24T23:48:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.285+0000 7efef3fff640 1 -- 172.21.15.16:0/790970314 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2475652516 0 0) 0x7efedc003520 con 0x7efef40729a0 2024-04-24T23:48:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.286+0000 7efef9595640 1 -- 172.21.15.16:0/790970314 >> v1:172.21.15.156:6789/0 conn(0x7efef406e910 legacy=0x7efef4072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.286+0000 7efef9595640 1 -- 172.21.15.16:0/790970314 >> v1:172.21.15.16:6789/0 conn(0x7efef40729a0 legacy=0x7efef40ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:49.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.286+0000 7efef9595640 1 -- 172.21.15.16:0/790970314 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef40cc400 con 0x7efef415a450 2024-04-24T23:48:49.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.286+0000 7efef9595640 1 --2- 172.21.15.16:0/790970314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef415a450 0x7efef40b2a20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efeec00a940 tx=0x7efeec02b4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5376a30dd5bca842 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:49.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.286+0000 7efef3fff640 1 -- 172.21.15.16:0/790970314 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee40019f0 con 0x7efef415a450 2024-04-24T23:48:49.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.286+0000 7efef3fff640 1 -- 172.21.15.16:0/790970314 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeec030510 con 0x7efef415a450 2024-04-24T23:48:49.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.286+0000 7efef3fff640 1 -- 172.21.15.16:0/790970314 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeec00b040 con 0x7efef415a450 2024-04-24T23:48:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.287+0000 7efefb7f9640 1 -- 172.21.15.16:0/790970314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef415a450 msgr2=0x7efef40b2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.287+0000 7efefb7f9640 1 --2- 172.21.15.16:0/790970314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef415a450 0x7efef40b2a20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efeec00a940 tx=0x7efeec02b4f0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.287+0000 7efefb7f9640 1 -- 172.21.15.16:0/790970314 shutdown_connections 2024-04-24T23:48:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.287+0000 7efefb7f9640 1 --2- 172.21.15.16:0/790970314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef415a450 0x7efef40b2a20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.287+0000 7efefb7f9640 1 -- 172.21.15.16:0/790970314 >> 172.21.15.16:0/790970314 conn(0x7efef40b2000 msgr2=0x7efef40cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.287+0000 7efefb7f9640 1 -- 172.21.15.16:0/790970314 shutdown_connections 2024-04-24T23:48:49.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.287+0000 7efefb7f9640 1 -- 172.21.15.16:0/790970314 wait complete. 2024-04-24T23:48:49.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.288+0000 7efefb7f9640 1 Processor -- start 2024-04-24T23:48:49.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.288+0000 7efefb7f9640 1 -- start start 2024-04-24T23:48:49.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efefb7f9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efef406e910 0x7efef41f7f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:49.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efefb7f9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef40729a0 0x7efef4145450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:49.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efefb7f9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef415a450 0x7efef415cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:49.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efefb7f9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efef40b30e0 con 0x7efef40729a0 2024-04-24T23:48:49.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efefb7f9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efef40b2f60 con 0x7efef415a450 2024-04-24T23:48:49.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efefb7f9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7efef4071a00 con 0x7efef406e910 2024-04-24T23:48:49.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efef8d94640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef40729a0 0x7efef4145450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:49.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efef8d94640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef40729a0 0x7efef4145450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43696/0 (socket says 172.21.15.16:43696) 2024-04-24T23:48:49.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efef8d94640 1 -- 172.21.15.16:0/1301535538 learned_addr learned my addr 172.21.15.16:0/1301535538 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:49.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efef9d96640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef415a450 0x7efef415cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:49.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efef9595640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efef406e910 0x7efef41f7f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:49.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efef8d94640 1 -- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efef406e910 msgr2=0x7efef41f7f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:49.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efef8d94640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efef406e910 0x7efef41f7f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efef8d94640 1 -- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef415a450 msgr2=0x7efef415cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:49.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efef8d94640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef415a450 0x7efef415cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efef8d94640 1 -- 172.21.15.16:0/1301535538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef41f8450 con 0x7efef40729a0 2024-04-24T23:48:49.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.289+0000 7efef9d96640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef415a450 0x7efef415cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:49.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.290+0000 7efef8d94640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef40729a0 0x7efef4145450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efee800c450 tx=0x7efee800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:49.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.290+0000 7efef1ffb640 1 -- 172.21.15.16:0/1301535538 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8017070 con 0x7efef40729a0 2024-04-24T23:48:49.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.290+0000 7efef1ffb640 1 -- 172.21.15.16:0/1301535538 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee80052d0 con 0x7efef40729a0 2024-04-24T23:48:49.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.290+0000 7efefb7f9640 1 -- 172.21.15.16:0/1301535538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef415ab60 con 0x7efef40729a0 2024-04-24T23:48:49.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.290+0000 7efef1ffb640 1 -- 172.21.15.16:0/1301535538 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8003830 con 0x7efef40729a0 2024-04-24T23:48:49.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.290+0000 7efefb7f9640 1 -- 172.21.15.16:0/1301535538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef41463b0 con 0x7efef40729a0 2024-04-24T23:48:49.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.291+0000 7efefb7f9640 1 -- 172.21.15.16:0/1301535538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efef40731f0 con 0x7efef40729a0 2024-04-24T23:48:49.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.291+0000 7efef1ffb640 1 -- 172.21.15.16:0/1301535538 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7efee801a070 con 0x7efef40729a0 2024-04-24T23:48:49.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.292+0000 7efef1ffb640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efee006be30 0x7efee006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:49.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.292+0000 7efef1ffb640 1 -- 172.21.15.16:0/1301535538 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7efee8089e80 con 0x7efef40729a0 2024-04-24T23:48:49.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.292+0000 7efef9595640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efee006be30 0x7efee006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:49.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.293+0000 7efef9595640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efee006be30 0x7efee006e2f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efeec035000 tx=0x7efeec00c040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:49.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.296+0000 7efef1ffb640 1 -- 172.21.15.16:0/1301535538 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7efef41f8450 con 0x7efef40729a0 2024-04-24T23:48:49.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.519+0000 7efefb7f9640 1 -- 172.21.15.16:0/1301535538 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efef415aea0 con 0x7efef40729a0 2024-04-24T23:48:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.519+0000 7efef1ffb640 1 -- 172.21.15.16:0/1301535538 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1392 (secure 0 0 0) 0x7efee8058c10 con 0x7efef40729a0 2024-04-24T23:48:49.519 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:48:49.519 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3530738236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3530738236},{"type":"v1","addr":"172.21.15.16:6835","nonce":3530738236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-24T23:48:49.520 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 2 2024-04-24T23:48:49.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.521+0000 7efefb7f9640 1 -- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efee006be30 msgr2=0x7efee006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.521+0000 7efefb7f9640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efee006be30 0x7efee006e2f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7efeec035000 tx=0x7efeec00c040 comp rx=0 tx=0).stop 2024-04-24T23:48:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.521+0000 7efefb7f9640 1 -- 172.21.15.16:0/1301535538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef40729a0 msgr2=0x7efef4145450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.521+0000 7efefb7f9640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef40729a0 0x7efef4145450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efee800c450 tx=0x7efee800c910 comp rx=0 tx=0).stop 2024-04-24T23:48:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.522+0000 7efefb7f9640 1 -- 172.21.15.16:0/1301535538 shutdown_connections 2024-04-24T23:48:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.522+0000 7efefb7f9640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efee006be30 0x7efee006e2f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.522+0000 7efefb7f9640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efef415a450 0x7efef415cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.522+0000 7efefb7f9640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efef40729a0 0x7efef4145450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.522+0000 7efefb7f9640 1 --2- 172.21.15.16:0/1301535538 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efef406e910 0x7efef41f7f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.522+0000 7efefb7f9640 1 -- 172.21.15.16:0/1301535538 >> 172.21.15.16:0/1301535538 conn(0x7efef40b2000 msgr2=0x7efef40ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.522+0000 7efefb7f9640 1 -- 172.21.15.16:0/1301535538 shutdown_connections 2024-04-24T23:48:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:49.522+0000 7efefb7f9640 1 -- 172.21.15.16:0/1301535538 wait complete. 2024-04-24T23:48:51.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:48:51.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.646+0000 7f6482b5a640 1 Processor -- start 2024-04-24T23:48:51.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.646+0000 7f6482b5a640 1 -- start start 2024-04-24T23:48:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.646+0000 7f6482b5a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f647c15e930 0x7f647c15ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.646+0000 7f6482b5a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f647c0cff10 0x7f647c06b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.646+0000 7f6482b5a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f647c0d0370 con 0x7f647c15dac0 2024-04-24T23:48:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.646+0000 7f6482b5a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f647c06bcd0 con 0x7f647c15e930 2024-04-24T23:48:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.646+0000 7f6482b5a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f647c06bfb0 con 0x7f647c0cff10 2024-04-24T23:48:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.647+0000 7f64808f6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f647c15e930 0x7f647c15ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.647+0000 7f64808f6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f647c15e930 0x7f647c15ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43708/0 (socket says 172.21.15.16:43708) 2024-04-24T23:48:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.647+0000 7f64808f6640 1 -- 172.21.15.16:0/2769393118 learned_addr learned my addr 172.21.15.16:0/2769393118 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.647+0000 7f647bfff640 1 --2- 172.21.15.16:0/2769393118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f647c0cff10 0x7f647c06b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.647+0000 7f64808f6640 1 -- 172.21.15.16:0/2769393118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f647c0cff10 msgr2=0x7f647c06b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:51.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.647+0000 7f64808f6640 1 --2- 172.21.15.16:0/2769393118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f647c0cff10 0x7f647c06b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.647+0000 7f64808f6640 1 -- 172.21.15.16:0/2769393118 >> v1:172.21.15.156:6789/0 conn(0x7f647c15dac0 legacy=0x7f647c15df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:51.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.647+0000 7f64808f6640 1 -- 172.21.15.16:0/2769393118 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f647c06c290 con 0x7f647c15e930 2024-04-24T23:48:51.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.648+0000 7f64808f6640 1 --2- 172.21.15.16:0/2769393118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c15e930 0x7f647c15ed10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6470009870 tx=0x7f647002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265a400b44dcced1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:51.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.648+0000 7f647b7fe640 1 -- 172.21.15.16:0/2769393118 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470037070 con 0x7f647c15e930 2024-04-24T23:48:51.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.648+0000 7f647b7fe640 1 -- 172.21.15.16:0/2769393118 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6470032440 con 0x7f647c15e930 2024-04-24T23:48:51.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.648+0000 7f647b7fe640 1 -- 172.21.15.16:0/2769393118 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470032810 con 0x7f647c15e930 2024-04-24T23:48:51.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.648+0000 7f6482b5a640 1 -- 172.21.15.16:0/2769393118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c15e930 msgr2=0x7f647c15ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:51.649 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.648+0000 7f6482b5a640 1 --2- 172.21.15.16:0/2769393118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c15e930 0x7f647c15ed10 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6470009870 tx=0x7f647002a210 comp rx=0 tx=0).stop 2024-04-24T23:48:51.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.648+0000 7f6482b5a640 1 -- 172.21.15.16:0/2769393118 shutdown_connections 2024-04-24T23:48:51.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.648+0000 7f6482b5a640 1 --2- 172.21.15.16:0/2769393118 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f647c0cff10 0x7f647c06b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.648+0000 7f6482b5a640 1 --2- 172.21.15.16:0/2769393118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c15e930 0x7f647c15ed10 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.648+0000 7f6482b5a640 1 -- 172.21.15.16:0/2769393118 >> 172.21.15.16:0/2769393118 conn(0x7f647c0d4050 msgr2=0x7f647c0d4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:51.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.648+0000 7f6482b5a640 1 -- 172.21.15.16:0/2769393118 shutdown_connections 2024-04-24T23:48:51.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.649+0000 7f6482b5a640 1 -- 172.21.15.16:0/2769393118 wait complete. 2024-04-24T23:48:51.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.650+0000 7f6482b5a640 1 Processor -- start 2024-04-24T23:48:51.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.650+0000 7f6482b5a640 1 -- start start 2024-04-24T23:48:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f6482b5a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f647c0cff10 0x7f647c15c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f6482b5a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f647c15dac0 0x7f647c14f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f6482b5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c15e930 0x7f647c0c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f6482b5a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f647c0d1bf0 con 0x7f647c15e930 2024-04-24T23:48:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f6482b5a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f647c06c850 con 0x7f647c0cff10 2024-04-24T23:48:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f6482b5a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f647c0d1d70 con 0x7f647c15dac0 2024-04-24T23:48:51.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f64810f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c15e930 0x7f647c0c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f64810f7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c15e930 0x7f647c0c8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43724/0 (socket says 172.21.15.16:43724) 2024-04-24T23:48:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f64810f7640 1 -- 172.21.15.16:0/1979680175 learned_addr learned my addr 172.21.15.16:0/1979680175 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f647bfff640 1 --2- 172.21.15.16:0/1979680175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f647c15dac0 0x7f647c14f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f64808f6640 1 --2- 172.21.15.16:0/1979680175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f647c0cff10 0x7f647c15c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f64810f7640 1 -- 172.21.15.16:0/1979680175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f647c15dac0 msgr2=0x7f647c14f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f64810f7640 1 --2- 172.21.15.16:0/1979680175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f647c15dac0 0x7f647c14f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f64810f7640 1 -- 172.21.15.16:0/1979680175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f647c0cff10 msgr2=0x7f647c15c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f64810f7640 1 --2- 172.21.15.16:0/1979680175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f647c0cff10 0x7f647c15c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f64810f7640 1 -- 172.21.15.16:0/1979680175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f647c0c9410 con 0x7f647c15e930 2024-04-24T23:48:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f64810f7640 1 --2- 172.21.15.16:0/1979680175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c15e930 0x7f647c0c8ed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f647400c7b0 tx=0x7f647400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f64797fa640 1 -- 172.21.15.16:0/1979680175 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474004b60 con 0x7f647c15e930 2024-04-24T23:48:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f64797fa640 1 -- 172.21.15.16:0/1979680175 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6474004d00 con 0x7f647c15e930 2024-04-24T23:48:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f64797fa640 1 -- 172.21.15.16:0/1979680175 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647400aa30 con 0x7f647c15e930 2024-04-24T23:48:51.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.651+0000 7f6482b5a640 1 -- 172.21.15.16:0/1979680175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f647c0c7e30 con 0x7f647c15e930 2024-04-24T23:48:51.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.652+0000 7f6482b5a640 1 -- 172.21.15.16:0/1979680175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f647c0c83f0 con 0x7f647c15e930 2024-04-24T23:48:51.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.653+0000 7f6482b5a640 1 -- 172.21.15.16:0/1979680175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f647c06dfe0 con 0x7f647c15e930 2024-04-24T23:48:51.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.653+0000 7f64797fa640 1 -- 172.21.15.16:0/1979680175 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f647400abd0 con 0x7f647c15e930 2024-04-24T23:48:51.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.654+0000 7f64797fa640 1 --2- 172.21.15.16:0/1979680175 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f646006be30 0x7f646006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:51.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.654+0000 7f64797fa640 1 -- 172.21.15.16:0/1979680175 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f647408acb0 con 0x7f647c15e930 2024-04-24T23:48:51.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.654+0000 7f64808f6640 1 --2- 172.21.15.16:0/1979680175 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f646006be30 0x7f646006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:51.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.654+0000 7f64808f6640 1 --2- 172.21.15.16:0/1979680175 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f646006be30 0x7f646006e2f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6470034000 tx=0x7f647002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:51.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.658+0000 7f64797fa640 1 -- 172.21.15.16:0/1979680175 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6474020070 con 0x7f647c15e930 2024-04-24T23:48:51.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.879+0000 7f6482b5a640 1 -- 172.21.15.16:0/1979680175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f647c04f5e0 con 0x7f647c15e930 2024-04-24T23:48:51.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.880+0000 7f64797fa640 1 -- 172.21.15.16:0/1979680175 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3867 (secure 0 0 0) 0x7f6474059a40 con 0x7f647c15e930 2024-04-24T23:48:51.880 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:48:51.880 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1623032341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1623032341},{"type":"v1","addr":"172.21.15.156:6836","nonce":1623032341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3530738236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3530738236},{"type":"v1","addr":"172.21.15.16:6835","nonce":3530738236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2153063131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2153063131},{"type":"v1","addr":"172.21.15.16:6834","nonce":2153063131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4294,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/289127714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":289127714},{"type":"v1","addr":"172.21.15.156:6837","nonce":289127714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-24T23:48:51.880 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 3 2024-04-24T23:48:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.882+0000 7f6482b5a640 1 -- 172.21.15.16:0/1979680175 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f646006be30 msgr2=0x7f646006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.882+0000 7f6482b5a640 1 --2- 172.21.15.16:0/1979680175 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f646006be30 0x7f646006e2f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6470034000 tx=0x7f647002e000 comp rx=0 tx=0).stop 2024-04-24T23:48:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.882+0000 7f6482b5a640 1 -- 172.21.15.16:0/1979680175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c15e930 msgr2=0x7f647c0c8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:51.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.882+0000 7f6482b5a640 1 --2- 172.21.15.16:0/1979680175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c15e930 0x7f647c0c8ed0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f647400c7b0 tx=0x7f647400cc70 comp rx=0 tx=0).stop 2024-04-24T23:48:51.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.882+0000 7f6482b5a640 1 -- 172.21.15.16:0/1979680175 shutdown_connections 2024-04-24T23:48:51.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.882+0000 7f6482b5a640 1 --2- 172.21.15.16:0/1979680175 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f646006be30 0x7f646006e2f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.882+0000 7f6482b5a640 1 --2- 172.21.15.16:0/1979680175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c15e930 0x7f647c0c8ed0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.882+0000 7f6482b5a640 1 --2- 172.21.15.16:0/1979680175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f647c15dac0 0x7f647c14f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.882+0000 7f6482b5a640 1 --2- 172.21.15.16:0/1979680175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f647c0cff10 0x7f647c15c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.882+0000 7f6482b5a640 1 -- 172.21.15.16:0/1979680175 >> 172.21.15.16:0/1979680175 conn(0x7f647c0d4050 msgr2=0x7f647c0d4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:51.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.882+0000 7f6482b5a640 1 -- 172.21.15.16:0/1979680175 shutdown_connections 2024-04-24T23:48:51.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.882+0000 7f6482b5a640 1 -- 172.21.15.16:0/1979680175 wait complete. 2024-04-24T23:48:51.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:48:51.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.993+0000 7f69a6d4f640 1 Processor -- start 2024-04-24T23:48:51.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.993+0000 7f69a6d4f640 1 -- start start 2024-04-24T23:48:51.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.994+0000 7f69a6d4f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f69a00b2bb0 0x7f69a00cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:51.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.994+0000 7f69a6d4f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69a00d1fa0 0x7f69a00d23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:51.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.994+0000 7f69a6d4f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69a00b3390 con 0x7f69a015e020 2024-04-24T23:48:51.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.994+0000 7f69a6d4f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f69a006bfc0 con 0x7f69a00d1fa0 2024-04-24T23:48:51.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.994+0000 7f69a6d4f640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f69a006c2a0 con 0x7f69a00b2bb0 2024-04-24T23:48:51.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.994+0000 7f69a52ec640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69a00d1fa0 0x7f69a00d23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:51.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.994+0000 7f69a52ec640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69a00d1fa0 0x7f69a00d23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43728/0 (socket says 172.21.15.16:43728) 2024-04-24T23:48:51.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.994+0000 7f69a52ec640 1 -- 172.21.15.16:0/1259174819 learned_addr learned my addr 172.21.15.16:0/1259174819 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:51.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.995+0000 7f699ffff640 1 --2- 172.21.15.16:0/1259174819 >> v2:172.21.15.156:3300/0 conn(0x7f69a00b2bb0 0x7f69a00cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:51.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.995+0000 7f69a52ec640 1 -- 172.21.15.16:0/1259174819 >> v2:172.21.15.156:3300/0 conn(0x7f69a00b2bb0 msgr2=0x7f69a00cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:51.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.995+0000 7f69a52ec640 1 --2- 172.21.15.16:0/1259174819 >> v2:172.21.15.156:3300/0 conn(0x7f69a00b2bb0 0x7f69a00cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.995+0000 7f69a52ec640 1 -- 172.21.15.16:0/1259174819 >> v1:172.21.15.156:6789/0 conn(0x7f69a015e020 legacy=0x7f69a00cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:51.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.995+0000 7f69a52ec640 1 -- 172.21.15.16:0/1259174819 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a006c580 con 0x7f69a00d1fa0 2024-04-24T23:48:51.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.995+0000 7f69a52ec640 1 --2- 172.21.15.16:0/1259174819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a00d1fa0 0x7f69a00d23b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f69a008d070 tx=0x7f699002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1414c72870269ca server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:51.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.995+0000 7f699f7fe640 1 -- 172.21.15.16:0/1259174819 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990037070 con 0x7f69a00d1fa0 2024-04-24T23:48:51.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.996+0000 7f699f7fe640 1 -- 172.21.15.16:0/1259174819 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6990032440 con 0x7f69a00d1fa0 2024-04-24T23:48:51.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.996+0000 7f699f7fe640 1 -- 172.21.15.16:0/1259174819 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990032810 con 0x7f69a00d1fa0 2024-04-24T23:48:51.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.996+0000 7f69a6d4f640 1 -- 172.21.15.16:0/1259174819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a00d1fa0 msgr2=0x7f69a00d23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:51.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.996+0000 7f69a6d4f640 1 --2- 172.21.15.16:0/1259174819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a00d1fa0 0x7f69a00d23b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f69a008d070 tx=0x7f699002a1a0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.996+0000 7f69a6d4f640 1 -- 172.21.15.16:0/1259174819 shutdown_connections 2024-04-24T23:48:51.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.996+0000 7f69a6d4f640 1 --2- 172.21.15.16:0/1259174819 >> v2:172.21.15.156:3300/0 conn(0x7f69a00b2bb0 0x7f69a00cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.996+0000 7f69a6d4f640 1 --2- 172.21.15.16:0/1259174819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a00d1fa0 0x7f69a00d23b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.996+0000 7f69a6d4f640 1 -- 172.21.15.16:0/1259174819 >> 172.21.15.16:0/1259174819 conn(0x7f69a00d3e10 msgr2=0x7f69a00d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.996+0000 7f69a6d4f640 1 -- 172.21.15.16:0/1259174819 shutdown_connections 2024-04-24T23:48:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.996+0000 7f69a6d4f640 1 -- 172.21.15.16:0/1259174819 wait complete. 2024-04-24T23:48:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.997+0000 7f69a6d4f640 1 Processor -- start 2024-04-24T23:48:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.997+0000 7f69a6d4f640 1 -- start start 2024-04-24T23:48:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.998+0000 7f69a6d4f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69a00b2bb0 0x7f69a01f48a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.998+0000 7f69a6d4f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a00d1fa0 0x7f69a00ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.998+0000 7f69a6d4f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69a015e020 0x7f69a00ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.998+0000 7f69a6d4f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f69a015e400 con 0x7f69a00d1fa0 2024-04-24T23:48:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.998+0000 7f69a6d4f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f69a00d1a80 con 0x7f69a015e020 2024-04-24T23:48:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.998+0000 7f69a6d4f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f69a00cfec0 con 0x7f69a00b2bb0 2024-04-24T23:48:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.998+0000 7f699ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a00d1fa0 0x7f69a00ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.998+0000 7f699ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a00d1fa0 0x7f69a00ca150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43742/0 (socket says 172.21.15.16:43742) 2024-04-24T23:48:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.998+0000 7f699ffff640 1 -- 172.21.15.16:0/117601201 learned_addr learned my addr 172.21.15.16:0/117601201 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.998+0000 7f69a52ec640 1 --2- 172.21.15.16:0/117601201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69a015e020 0x7f69a00ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.998+0000 7f69a4aeb640 1 --2- 172.21.15.16:0/117601201 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69a00b2bb0 0x7f69a01f48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.999+0000 7f699ffff640 1 -- 172.21.15.16:0/117601201 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69a00b2bb0 msgr2=0x7f69a01f48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.999+0000 7f699ffff640 1 --2- 172.21.15.16:0/117601201 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69a00b2bb0 0x7f69a01f48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.999+0000 7f699ffff640 1 -- 172.21.15.16:0/117601201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69a015e020 msgr2=0x7f69a00ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.999+0000 7f699ffff640 1 --2- 172.21.15.16:0/117601201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69a015e020 0x7f69a00ccef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.999+0000 7f699ffff640 1 -- 172.21.15.16:0/117601201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a00cd430 con 0x7f69a00d1fa0 2024-04-24T23:48:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.999+0000 7f699ffff640 1 --2- 172.21.15.16:0/117601201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a00d1fa0 0x7f69a00ca150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f699400c820 tx=0x7f699400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.999+0000 7f699d7fa640 1 -- 172.21.15.16:0/117601201 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6994015070 con 0x7f69a00d1fa0 2024-04-24T23:48:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.999+0000 7f699d7fa640 1 -- 172.21.15.16:0/117601201 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6994004240 con 0x7f69a00d1fa0 2024-04-24T23:48:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.999+0000 7f69a6d4f640 1 -- 172.21.15.16:0/117601201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69a00cd6d0 con 0x7f69a00d1fa0 2024-04-24T23:48:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.999+0000 7f699d7fa640 1 -- 172.21.15.16:0/117601201 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f699400ce10 con 0x7f69a00d1fa0 2024-04-24T23:48:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:51.999+0000 7f69a6d4f640 1 -- 172.21.15.16:0/117601201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69a01f40a0 con 0x7f69a00d1fa0 2024-04-24T23:48:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.000+0000 7f69a6d4f640 1 -- 172.21.15.16:0/117601201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69a006dfe0 con 0x7f69a00d1fa0 2024-04-24T23:48:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.000+0000 7f699d7fa640 1 -- 172.21.15.16:0/117601201 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f69940077d0 con 0x7f69a00d1fa0 2024-04-24T23:48:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.001+0000 7f699d7fa640 1 --2- 172.21.15.16:0/117601201 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f698c06be30 0x7f698c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.001+0000 7f699d7fa640 1 -- 172.21.15.16:0/117601201 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f699408a8f0 con 0x7f69a00d1fa0 2024-04-24T23:48:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.001+0000 7f69a4aeb640 1 --2- 172.21.15.16:0/117601201 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f698c06be30 0x7f698c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.001+0000 7f69a4aeb640 1 --2- 172.21.15.16:0/117601201 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f698c06be30 0x7f698c06e2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f699800b450 tx=0x7f699800d040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:52.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.005+0000 7f699d7fa640 1 -- 172.21.15.16:0/117601201 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6994058fa0 con 0x7f69a00d1fa0 2024-04-24T23:48:52.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.189+0000 7f69a6d4f640 1 -- 172.21.15.16:0/117601201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69a004f5e0 con 0x7f69a00d1fa0 2024-04-24T23:48:52.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.189+0000 7f699d7fa640 1 -- 172.21.15.16:0/117601201 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12872 (secure 0 0 0) 0x7f699405d2a0 con 0x7f69a00d1fa0 2024-04-24T23:48:52.189 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:48:52.189 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:45.450145+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:48:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.191+0000 7f69a6d4f640 1 -- 172.21.15.16:0/117601201 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f698c06be30 msgr2=0x7f698c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.191+0000 7f69a6d4f640 1 --2- 172.21.15.16:0/117601201 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f698c06be30 0x7f698c06e2f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f699800b450 tx=0x7f699800d040 comp rx=0 tx=0).stop 2024-04-24T23:48:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.191+0000 7f69a6d4f640 1 -- 172.21.15.16:0/117601201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a00d1fa0 msgr2=0x7f69a00ca150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.191+0000 7f69a6d4f640 1 --2- 172.21.15.16:0/117601201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a00d1fa0 0x7f69a00ca150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f699400c820 tx=0x7f699400cce0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.192+0000 7f69a6d4f640 1 -- 172.21.15.16:0/117601201 shutdown_connections 2024-04-24T23:48:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.192+0000 7f69a6d4f640 1 --2- 172.21.15.16:0/117601201 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f698c06be30 0x7f698c06e2f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.192+0000 7f69a6d4f640 1 --2- 172.21.15.16:0/117601201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69a015e020 0x7f69a00ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.192+0000 7f69a6d4f640 1 --2- 172.21.15.16:0/117601201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69a00d1fa0 0x7f69a00ca150 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.192+0000 7f69a6d4f640 1 --2- 172.21.15.16:0/117601201 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f69a00b2bb0 0x7f69a01f48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.192+0000 7f69a6d4f640 1 -- 172.21.15.16:0/117601201 >> 172.21.15.16:0/117601201 conn(0x7f69a00d3e10 msgr2=0x7f69a00b6fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.192+0000 7f69a6d4f640 1 -- 172.21.15.16:0/117601201 shutdown_connections 2024-04-24T23:48:52.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.192+0000 7f69a6d4f640 1 -- 172.21.15.16:0/117601201 wait complete. 2024-04-24T23:48:52.202 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T23:48:52.202 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:48:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.305+0000 7f261bed2640 1 Processor -- start 2024-04-24T23:48:52.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.306+0000 7f261bed2640 1 -- start start 2024-04-24T23:48:52.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.306+0000 7f261bed2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f26141478d0 0x7f26140d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.306+0000 7f261bed2640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26140b2a10 con 0x7f26140d2090 2024-04-24T23:48:52.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.306+0000 7f261bed2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26140b2cf0 con 0x7f26140b23a0 2024-04-24T23:48:52.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.306+0000 7f261bed2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f26140d24f0 con 0x7f26141478d0 2024-04-24T23:48:52.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f261946d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f26140d2090 0x7f26140b1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60734/0 (socket says 172.21.15.16:60734) 2024-04-24T23:48:52.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f261946d640 1 -- 172.21.15.16:0/1751894318 learned_addr learned my addr 172.21.15.16:0/1751894318 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:48:52.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2619c6e640 1 --2- 172.21.15.16:0/1751894318 >> v2:172.21.15.156:3300/0 conn(0x7f26141478d0 0x7f26140d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 -- 172.21.15.16:0/1751894318 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3357571144 0 0) 0x7f26140b2a10 con 0x7f26140d2090 2024-04-24T23:48:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 -- 172.21.15.16:0/1751894318 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f25fc003520 con 0x7f26140d2090 2024-04-24T23:48:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 -- 172.21.15.16:0/1751894318 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1006979978 0 0) 0x7f25fc003520 con 0x7f26140d2090 2024-04-24T23:48:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 -- 172.21.15.16:0/1751894318 >> v2:172.21.15.156:3300/0 conn(0x7f26141478d0 msgr2=0x7f26140d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 --2- 172.21.15.16:0/1751894318 >> v2:172.21.15.156:3300/0 conn(0x7f26141478d0 0x7f26140d1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 -- 172.21.15.16:0/1751894318 >> v1:172.21.15.156:6789/0 conn(0x7f26140b23a0 legacy=0x7f26140b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 -- 172.21.15.16:0/1751894318 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26140b2fd0 con 0x7f26140d2090 2024-04-24T23:48:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 -- 172.21.15.16:0/1751894318 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f2608001670 con 0x7f26140d2090 2024-04-24T23:48:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 -- 172.21.15.16:0/1751894318 >> v1:172.21.15.16:6789/0 conn(0x7f26140d2090 legacy=0x7f26140b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 --2- 172.21.15.16:0/1751894318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25fc0039b0 0x7f25fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 --2- 172.21.15.16:0/1751894318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25fc004800 0x7f25fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 --2- 172.21.15.16:0/1751894318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f25fc0051c0 0x7f25fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 -- 172.21.15.16:0/1751894318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc00c030 con 0x7f25fc0039b0 2024-04-24T23:48:52.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 -- 172.21.15.16:0/1751894318 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f25fc00c1e0 con 0x7f25fc004800 2024-04-24T23:48:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2618c6c640 1 -- 172.21.15.16:0/1751894318 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f25fc00c3e0 con 0x7f25fc0051c0 2024-04-24T23:48:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2619c6e640 1 --2- 172.21.15.16:0/1751894318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25fc0039b0 0x7f25fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f261946d640 1 --2- 172.21.15.16:0/1751894318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25fc004800 0x7f25fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f261a46f640 1 --2- 172.21.15.16:0/1751894318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f25fc0051c0 0x7f25fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.307+0000 7f2619c6e640 1 -- 172.21.15.16:0/1751894318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f25fc0051c0 msgr2=0x7f25fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.308+0000 7f2619c6e640 1 --2- 172.21.15.16:0/1751894318 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f25fc0051c0 0x7f25fc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.308+0000 7f2619c6e640 1 -- 172.21.15.16:0/1751894318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25fc004800 msgr2=0x7f25fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.308+0000 7f2619c6e640 1 --2- 172.21.15.16:0/1751894318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25fc004800 0x7f25fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.308+0000 7f2619c6e640 1 -- 172.21.15.16:0/1751894318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25fc003520 con 0x7f25fc0039b0 2024-04-24T23:48:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.308+0000 7f261946d640 1 --2- 172.21.15.16:0/1751894318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25fc004800 0x7f25fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.308+0000 7f261a46f640 1 -- 172.21.15.16:0/1751894318 reap_dead start 2024-04-24T23:48:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.308+0000 7f2619c6e640 1 --2- 172.21.15.16:0/1751894318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25fc0039b0 0x7f25fc003dd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2604009fc0 tx=0x7f25fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2536677d18af8a9a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.308+0000 7f2618c6c640 1 -- 172.21.15.16:0/1751894318 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604034020 con 0x7f25fc0039b0 2024-04-24T23:48:52.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.308+0000 7f2618c6c640 1 -- 172.21.15.16:0/1751894318 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26040305f0 con 0x7f25fc0039b0 2024-04-24T23:48:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.308+0000 7f2618c6c640 1 -- 172.21.15.16:0/1751894318 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2604009c10 con 0x7f25fc0039b0 2024-04-24T23:48:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.309+0000 7f261bed2640 1 -- 172.21.15.16:0/1751894318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25fc0039b0 msgr2=0x7f25fc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.309+0000 7f261bed2640 1 --2- 172.21.15.16:0/1751894318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25fc0039b0 0x7f25fc003dd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2604009fc0 tx=0x7f25fc005830 comp rx=0 tx=0).stop 2024-04-24T23:48:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.309+0000 7f261bed2640 1 -- 172.21.15.16:0/1751894318 shutdown_connections 2024-04-24T23:48:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.309+0000 7f261bed2640 1 --2- 172.21.15.16:0/1751894318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25fc0039b0 0x7f25fc003dd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.309+0000 7f261bed2640 1 -- 172.21.15.16:0/1751894318 >> 172.21.15.16:0/1751894318 conn(0x7f261406b2c0 msgr2=0x7f2614072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.309+0000 7f261bed2640 1 -- 172.21.15.16:0/1751894318 shutdown_connections 2024-04-24T23:48:52.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.309+0000 7f261bed2640 1 -- 172.21.15.16:0/1751894318 wait complete. 2024-04-24T23:48:52.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.310+0000 7f261bed2640 1 Processor -- start 2024-04-24T23:48:52.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.310+0000 7f261bed2640 1 -- start start 2024-04-24T23:48:52.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261bed2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25fc0039b0 0x7f2614144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261bed2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f261415d280 0x7f261415d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261bed2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f261413ee60 0x7f261413f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261bed2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f26140d1b50 con 0x7f261415d280 2024-04-24T23:48:52.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261bed2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f26140b3620 con 0x7f25fc0039b0 2024-04-24T23:48:52.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261bed2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f26140d1cd0 con 0x7f261413ee60 2024-04-24T23:48:52.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261946d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f261415d280 0x7f261415d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261946d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f261415d280 0x7f261415d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43758/0 (socket says 172.21.15.16:43758) 2024-04-24T23:48:52.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261946d640 1 -- 172.21.15.16:0/2727214404 learned_addr learned my addr 172.21.15.16:0/2727214404 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:52.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261a46f640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f261413ee60 0x7f261413f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f2619c6e640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25fc0039b0 0x7f2614144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261946d640 1 -- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f261413ee60 msgr2=0x7f261413f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261946d640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f261413ee60 0x7f261413f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261946d640 1 -- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25fc0039b0 msgr2=0x7f2614144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261946d640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25fc0039b0 0x7f2614144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261946d640 1 -- 172.21.15.16:0/2727214404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f261413fb20 con 0x7f261415d280 2024-04-24T23:48:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f2619c6e640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25fc0039b0 0x7f2614144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.311+0000 7f261946d640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f261415d280 0x7f261415d700 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f260800e730 tx=0x7f260800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.312+0000 7f26127fc640 1 -- 172.21.15.16:0/2727214404 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260800ee40 con 0x7f261415d280 2024-04-24T23:48:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.312+0000 7f26127fc640 1 -- 172.21.15.16:0/2727214404 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2608018070 con 0x7f261415d280 2024-04-24T23:48:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.312+0000 7f261bed2640 1 -- 172.21.15.16:0/2727214404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f261413fe00 con 0x7f261415d280 2024-04-24T23:48:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.312+0000 7f26127fc640 1 -- 172.21.15.16:0/2727214404 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f260800cad0 con 0x7f261415d280 2024-04-24T23:48:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.312+0000 7f261bed2640 1 -- 172.21.15.16:0/2727214404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26141403c0 con 0x7f261415d280 2024-04-24T23:48:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.313+0000 7f261bed2640 1 -- 172.21.15.16:0/2727214404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2614068f90 con 0x7f261415d280 2024-04-24T23:48:52.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.313+0000 7f26127fc640 1 -- 172.21.15.16:0/2727214404 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f260800cc70 con 0x7f261415d280 2024-04-24T23:48:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.313+0000 7f26127fc640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f260006be30 0x7f260006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.313+0000 7f26127fc640 1 -- 172.21.15.16:0/2727214404 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f260808a910 con 0x7f261415d280 2024-04-24T23:48:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.313+0000 7f2619c6e640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f260006be30 0x7f260006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.314+0000 7f2619c6e640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f260006be30 0x7f260006e2f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2604035000 tx=0x7f260402f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:52.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.318+0000 7f26127fc640 1 -- 172.21.15.16:0/2727214404 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f26080596a0 con 0x7f261415d280 2024-04-24T23:48:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.501+0000 7f261bed2640 1 -- 172.21.15.16:0/2727214404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f261413f830 con 0x7f261415d280 2024-04-24T23:48:52.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.501+0000 7f26127fc640 1 -- 172.21.15.16:0/2727214404 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) ==== 93+0+10 (secure 0 0 0) 0x7f2608058fa0 con 0x7f261415d280 2024-04-24T23:48:52.501 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:48:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.503+0000 7f261bed2640 1 -- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f260006be30 msgr2=0x7f260006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.503+0000 7f261bed2640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f260006be30 0x7f260006e2f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2604035000 tx=0x7f260402f000 comp rx=0 tx=0).stop 2024-04-24T23:48:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.503+0000 7f261bed2640 1 -- 172.21.15.16:0/2727214404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f261415d280 msgr2=0x7f261415d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.503+0000 7f261bed2640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f261415d280 0x7f261415d700 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f260800e730 tx=0x7f260800ebf0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.503+0000 7f261bed2640 1 -- 172.21.15.16:0/2727214404 shutdown_connections 2024-04-24T23:48:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.503+0000 7f261bed2640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f260006be30 0x7f260006e2f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.503+0000 7f261bed2640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f261413ee60 0x7f261413f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.503+0000 7f261bed2640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f261415d280 0x7f261415d700 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.503+0000 7f261bed2640 1 --2- 172.21.15.16:0/2727214404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25fc0039b0 0x7f2614144e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.503+0000 7f261bed2640 1 -- 172.21.15.16:0/2727214404 >> 172.21.15.16:0/2727214404 conn(0x7f261406b2c0 msgr2=0x7f26140b54d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.503+0000 7f261bed2640 1 -- 172.21.15.16:0/2727214404 shutdown_connections 2024-04-24T23:48:52.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.503+0000 7f261bed2640 1 -- 172.21.15.16:0/2727214404 wait complete. 2024-04-24T23:48:52.514 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-24T23:48:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.614+0000 7f18688f5640 1 Processor -- start 2024-04-24T23:48:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.614+0000 7f18688f5640 1 -- start start 2024-04-24T23:48:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f18688f5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f186415dfa0 0x7f18640d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f18688f5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18640ce5f0 0x7f18640cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f18688f5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f18640b5f00 0x7f18640b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f18688f5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f18640b7360 con 0x7f18640ce5f0 2024-04-24T23:48:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f18688f5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f18640cc640 con 0x7f18640b5f00 2024-04-24T23:48:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f18688f5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f18640d05a0 con 0x7f186415dfa0 2024-04-24T23:48:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f1861d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18640ce5f0 0x7f18640cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f1861d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18640ce5f0 0x7f18640cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43760/0 (socket says 172.21.15.16:43760) 2024-04-24T23:48:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f1861d9b640 1 -- 172.21.15.16:0/1987690016 learned_addr learned my addr 172.21.15.16:0/1987690016 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f186259c640 1 --2- 172.21.15.16:0/1987690016 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f186415dfa0 0x7f18640d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f1861d9b640 1 -- 172.21.15.16:0/1987690016 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f186415dfa0 msgr2=0x7f18640d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f1861d9b640 1 --2- 172.21.15.16:0/1987690016 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f186415dfa0 0x7f18640d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f1861d9b640 1 -- 172.21.15.16:0/1987690016 >> v2:172.21.15.156:3300/0 conn(0x7f18640b5f00 msgr2=0x7f18640b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f1862d9d640 1 --2- 172.21.15.16:0/1987690016 >> v2:172.21.15.156:3300/0 conn(0x7f18640b5f00 0x7f18640b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f1861d9b640 1 --2- 172.21.15.16:0/1987690016 >> v2:172.21.15.156:3300/0 conn(0x7f18640b5f00 0x7f18640b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f1861d9b640 1 -- 172.21.15.16:0/1987690016 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18640d0880 con 0x7f18640ce5f0 2024-04-24T23:48:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.615+0000 7f1862d9d640 1 --2- 172.21.15.16:0/1987690016 >> v2:172.21.15.156:3300/0 conn(0x7f18640b5f00 0x7f18640b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.616+0000 7f1861d9b640 1 --2- 172.21.15.16:0/1987690016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18640ce5f0 0x7f18640cea70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f18640cc1e0 tx=0x7f185802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70379f9166bd4fe9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.616+0000 7f186159a640 1 -- 172.21.15.16:0/1987690016 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1858037070 con 0x7f18640ce5f0 2024-04-24T23:48:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.616+0000 7f186159a640 1 -- 172.21.15.16:0/1987690016 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1858032440 con 0x7f18640ce5f0 2024-04-24T23:48:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.616+0000 7f186159a640 1 -- 172.21.15.16:0/1987690016 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1858032810 con 0x7f18640ce5f0 2024-04-24T23:48:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.617+0000 7f18688f5640 1 -- 172.21.15.16:0/1987690016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18640ce5f0 msgr2=0x7f18640cea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.617+0000 7f18688f5640 1 --2- 172.21.15.16:0/1987690016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18640ce5f0 0x7f18640cea70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f18640cc1e0 tx=0x7f185802a1e0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.617+0000 7f18688f5640 1 -- 172.21.15.16:0/1987690016 shutdown_connections 2024-04-24T23:48:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.617+0000 7f18688f5640 1 --2- 172.21.15.16:0/1987690016 >> v2:172.21.15.156:3300/0 conn(0x7f18640b5f00 0x7f18640b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.617+0000 7f18688f5640 1 --2- 172.21.15.16:0/1987690016 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18640ce5f0 0x7f18640cea70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.617+0000 7f18688f5640 1 --2- 172.21.15.16:0/1987690016 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f186415dfa0 0x7f18640d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.617+0000 7f18688f5640 1 -- 172.21.15.16:0/1987690016 >> 172.21.15.16:0/1987690016 conn(0x7f186406b4d0 msgr2=0x7f18640b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.617+0000 7f18688f5640 1 -- 172.21.15.16:0/1987690016 shutdown_connections 2024-04-24T23:48:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.617+0000 7f18688f5640 1 -- 172.21.15.16:0/1987690016 wait complete. 2024-04-24T23:48:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.618+0000 7f18688f5640 1 Processor -- start 2024-04-24T23:48:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.618+0000 7f18688f5640 1 -- start start 2024-04-24T23:48:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f18688f5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18640b5f00 0x7f186406d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f18688f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18640ce5f0 0x7f186406d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f18688f5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f186415dfa0 0x7f186406f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f18688f5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f186415e380 con 0x7f18640ce5f0 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f18688f5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f18640cc2d0 con 0x7f18640b5f00 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f18688f5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f18640d1520 con 0x7f186415dfa0 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f1861d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18640ce5f0 0x7f186406d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f1861d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18640ce5f0 0x7f186406d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43770/0 (socket says 172.21.15.16:43770) 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f1861d9b640 1 -- 172.21.15.16:0/705229482 learned_addr learned my addr 172.21.15.16:0/705229482 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f1862d9d640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f186415dfa0 0x7f186406f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f1861d9b640 1 -- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f186415dfa0 msgr2=0x7f186406f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f186259c640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18640b5f00 0x7f186406d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f1861d9b640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f186415dfa0 0x7f186406f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f1861d9b640 1 -- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18640b5f00 msgr2=0x7f186406d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f1861d9b640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18640b5f00 0x7f186406d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f1861d9b640 1 -- 172.21.15.16:0/705229482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18641f4770 con 0x7f18640ce5f0 2024-04-24T23:48:52.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f1862d9d640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f186415dfa0 0x7f186406f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f186259c640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18640b5f00 0x7f186406d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f1861d9b640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18640ce5f0 0x7f186406d9b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1858000c00 tx=0x7f18580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.619+0000 7f1852ffd640 1 -- 172.21.15.16:0/705229482 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1858041020 con 0x7f18640ce5f0 2024-04-24T23:48:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.620+0000 7f1852ffd640 1 -- 172.21.15.16:0/705229482 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18580306c0 con 0x7f18640ce5f0 2024-04-24T23:48:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.620+0000 7f18688f5640 1 -- 172.21.15.16:0/705229482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f186406ffe0 con 0x7f18640ce5f0 2024-04-24T23:48:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.620+0000 7f1852ffd640 1 -- 172.21.15.16:0/705229482 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1858037040 con 0x7f18640ce5f0 2024-04-24T23:48:52.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.620+0000 7f18688f5640 1 -- 172.21.15.16:0/705229482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18640705a0 con 0x7f18640ce5f0 2024-04-24T23:48:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.621+0000 7f18688f5640 1 -- 172.21.15.16:0/705229482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f186406bea0 con 0x7f18640ce5f0 2024-04-24T23:48:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.621+0000 7f1852ffd640 1 -- 172.21.15.16:0/705229482 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1858044020 con 0x7f18640ce5f0 2024-04-24T23:48:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.621+0000 7f1852ffd640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f184406be30 0x7f184406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.621+0000 7f1852ffd640 1 -- 172.21.15.16:0/705229482 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f18580ab3d0 con 0x7f18640ce5f0 2024-04-24T23:48:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.622+0000 7f186259c640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f184406be30 0x7f184406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.622+0000 7f186259c640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f184406be30 0x7f184406e2f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1854006fd0 tx=0x7f1854008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:52.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.626+0000 7f1852ffd640 1 -- 172.21.15.16:0/705229482 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f185804b030 con 0x7f18640ce5f0 2024-04-24T23:48:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.814+0000 7f18688f5640 1 -- 172.21.15.16:0/705229482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f18640c4e40 con 0x7f18640ce5f0 2024-04-24T23:48:52.814 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.814+0000 7f1852ffd640 1 -- 172.21.15.16:0/705229482 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7f185807a050 con 0x7f18640ce5f0 2024-04-24T23:48:52.814 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:48:52.814 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-04-24T23:48:52.814 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:48:52.814 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-04-24T23:48:52.814 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-04-24T23:48:52.814 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:48:52.815 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-04-24T23:48:52.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.816+0000 7f18688f5640 1 -- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f184406be30 msgr2=0x7f184406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.816+0000 7f18688f5640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f184406be30 0x7f184406e2f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1854006fd0 tx=0x7f1854008040 comp rx=0 tx=0).stop 2024-04-24T23:48:52.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.816+0000 7f18688f5640 1 -- 172.21.15.16:0/705229482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18640ce5f0 msgr2=0x7f186406d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.816+0000 7f18688f5640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18640ce5f0 0x7f186406d9b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1858000c00 tx=0x7f18580046c0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.816+0000 7f18688f5640 1 -- 172.21.15.16:0/705229482 shutdown_connections 2024-04-24T23:48:52.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.816+0000 7f18688f5640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f184406be30 0x7f184406e2f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.816+0000 7f18688f5640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f186415dfa0 0x7f186406f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.816+0000 7f18688f5640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18640ce5f0 0x7f186406d9b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.816+0000 7f18688f5640 1 --2- 172.21.15.16:0/705229482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18640b5f00 0x7f186406d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.816+0000 7f18688f5640 1 -- 172.21.15.16:0/705229482 >> 172.21.15.16:0/705229482 conn(0x7f186406b4d0 msgr2=0x7f18641607f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:52.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.816+0000 7f18688f5640 1 -- 172.21.15.16:0/705229482 shutdown_connections 2024-04-24T23:48:52.816 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.816+0000 7f18688f5640 1 -- 172.21.15.16:0/705229482 wait complete. 2024-04-24T23:48:52.826 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-24T23:48:52.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-24T23:48:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.931+0000 7fa8189b7640 1 Processor -- start 2024-04-24T23:48:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.931+0000 7fa8189b7640 1 -- start start 2024-04-24T23:48:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.932+0000 7fa8189b7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa81406bcb0 0x7fa81406c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.932+0000 7fa8189b7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8140d1aa0 con 0x7fa8140d2260 2024-04-24T23:48:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.932+0000 7fa8189b7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa81406b470 con 0x7fa81415dfb0 2024-04-24T23:48:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.932+0000 7fa8189b7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa8140d02b0 con 0x7fa81406bcb0 2024-04-24T23:48:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.932+0000 7fa812d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa8140d2260 0x7fa8140d26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60746/0 (socket says 172.21.15.16:60746) 2024-04-24T23:48:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.932+0000 7fa812d9d640 1 -- 172.21.15.16:0/2761364128 learned_addr learned my addr 172.21.15.16:0/2761364128 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:48:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.932+0000 7fa811d9b640 1 --2- 172.21.15.16:0/2761364128 >> v2:172.21.15.156:3300/0 conn(0x7fa81406bcb0 0x7fa81406c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.933+0000 7fa81159a640 1 -- 172.21.15.16:0/2761364128 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3517145416 0 0) 0x7fa8140d1aa0 con 0x7fa8140d2260 2024-04-24T23:48:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.933+0000 7fa81159a640 1 -- 172.21.15.16:0/2761364128 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa800003520 con 0x7fa8140d2260 2024-04-24T23:48:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.933+0000 7fa811d9b640 1 -- 172.21.15.16:0/2761364128 >> v1:172.21.15.156:6789/0 conn(0x7fa81415dfb0 legacy=0x7fa8140cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.933+0000 7fa811d9b640 1 -- 172.21.15.16:0/2761364128 >> v1:172.21.15.16:6789/0 conn(0x7fa8140d2260 legacy=0x7fa8140d26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.933+0000 7fa811d9b640 1 -- 172.21.15.16:0/2761364128 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa81406b730 con 0x7fa81406bcb0 2024-04-24T23:48:52.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.934+0000 7fa811d9b640 1 --2- 172.21.15.16:0/2761364128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa81406bcb0 0x7fa81406c110 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0097a0 tx=0x7fa7fc02a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fe3119449a47190 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:52.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.934+0000 7fa81159a640 1 -- 172.21.15.16:0/2761364128 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc037070 con 0x7fa81406bcb0 2024-04-24T23:48:52.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.934+0000 7fa81159a640 1 -- 172.21.15.16:0/2761364128 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7fc032440 con 0x7fa81406bcb0 2024-04-24T23:48:52.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.934+0000 7fa81159a640 1 -- 172.21.15.16:0/2761364128 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc0327f0 con 0x7fa81406bcb0 2024-04-24T23:48:52.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.934+0000 7fa8189b7640 1 -- 172.21.15.16:0/2761364128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa81406bcb0 msgr2=0x7fa81406c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.934+0000 7fa8189b7640 1 --2- 172.21.15.16:0/2761364128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa81406bcb0 0x7fa81406c110 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0097a0 tx=0x7fa7fc02a230 comp rx=0 tx=0).stop 2024-04-24T23:48:52.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.934+0000 7fa8189b7640 1 -- 172.21.15.16:0/2761364128 shutdown_connections 2024-04-24T23:48:52.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.934+0000 7fa8189b7640 1 --2- 172.21.15.16:0/2761364128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa81406bcb0 0x7fa81406c110 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.934+0000 7fa8189b7640 1 -- 172.21.15.16:0/2761364128 >> 172.21.15.16:0/2761364128 conn(0x7fa8140d3e30 msgr2=0x7fa8140d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:52.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.934+0000 7fa8189b7640 1 -- 172.21.15.16:0/2761364128 shutdown_connections 2024-04-24T23:48:52.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.934+0000 7fa8189b7640 1 -- 172.21.15.16:0/2761364128 wait complete. 2024-04-24T23:48:52.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.936+0000 7fa8189b7640 1 Processor -- start 2024-04-24T23:48:52.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.936+0000 7fa8189b7640 1 -- start start 2024-04-24T23:48:52.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.936+0000 7fa8189b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa81406bcb0 0x7fa8141f6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.936+0000 7fa8189b7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8140d2260 0x7fa8140b5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa8189b7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa81415dfb0 0x7fa8140b5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa8189b7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa8140d1dd0 con 0x7fa81406bcb0 2024-04-24T23:48:52.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa8189b7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa81406ba10 con 0x7fa8140d2260 2024-04-24T23:48:52.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa8189b7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fa8140b6f60 con 0x7fa81415dfb0 2024-04-24T23:48:52.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa81259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa81406bcb0 0x7fa8141f6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa81259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa81406bcb0 0x7fa8141f6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43784/0 (socket says 172.21.15.16:43784) 2024-04-24T23:48:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa81259c640 1 -- 172.21.15.16:0/2932006520 learned_addr learned my addr 172.21.15.16:0/2932006520 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa811d9b640 1 --2- 172.21.15.16:0/2932006520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8140d2260 0x7fa8140b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa812d9d640 1 --2- 172.21.15.16:0/2932006520 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa81415dfb0 0x7fa8140b5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa81259c640 1 -- 172.21.15.16:0/2932006520 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa81415dfb0 msgr2=0x7fa8140b5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa81259c640 1 --2- 172.21.15.16:0/2932006520 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa81415dfb0 0x7fa8140b5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa81259c640 1 -- 172.21.15.16:0/2932006520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8140d2260 msgr2=0x7fa8140b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa81259c640 1 --2- 172.21.15.16:0/2932006520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8140d2260 0x7fa8140b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa81259c640 1 -- 172.21.15.16:0/2932006520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8141e8760 con 0x7fa81406bcb0 2024-04-24T23:48:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa81259c640 1 --2- 172.21.15.16:0/2932006520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa81406bcb0 0x7fa8141f6860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00c430 tx=0x7fa80c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa80affd640 1 -- 172.21.15.16:0/2932006520 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c017070 con 0x7fa81406bcb0 2024-04-24T23:48:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa80affd640 1 -- 172.21.15.16:0/2932006520 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa80c0052d0 con 0x7fa81406bcb0 2024-04-24T23:48:52.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa8189b7640 1 -- 172.21.15.16:0/2932006520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8140cb3e0 con 0x7fa81406bcb0 2024-04-24T23:48:52.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa80affd640 1 -- 172.21.15.16:0/2932006520 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80c003830 con 0x7fa81406bcb0 2024-04-24T23:48:52.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.937+0000 7fa8189b7640 1 -- 172.21.15.16:0/2932006520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8140cb020 con 0x7fa81406bcb0 2024-04-24T23:48:52.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.938+0000 7fa8189b7640 1 -- 172.21.15.16:0/2932006520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa814068f90 con 0x7fa81406bcb0 2024-04-24T23:48:52.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.939+0000 7fa80affd640 1 -- 172.21.15.16:0/2932006520 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa80c01a070 con 0x7fa81406bcb0 2024-04-24T23:48:52.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.939+0000 7fa80affd640 1 --2- 172.21.15.16:0/2932006520 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa7f406be30 0x7fa7f406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:52.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.939+0000 7fa80affd640 1 -- 172.21.15.16:0/2932006520 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fa80c089dd0 con 0x7fa81406bcb0 2024-04-24T23:48:52.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.939+0000 7fa811d9b640 1 --2- 172.21.15.16:0/2932006520 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa7f406be30 0x7fa7f406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:52.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.940+0000 7fa811d9b640 1 --2- 172.21.15.16:0/2932006520 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa7f406be30 0x7fa7f406e2f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc034000 tx=0x7fa7fc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:52.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:52.943+0000 7fa80affd640 1 -- 172.21.15.16:0/2932006520 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa80c058480 con 0x7fa81406bcb0 2024-04-24T23:48:53.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.126+0000 7fa8189b7640 1 -- 172.21.15.16:0/2932006520 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fa8140cecb0 con 0x7fa81406bcb0 2024-04-24T23:48:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.469+0000 7fa80affd640 1 -- 172.21.15.16:0/2932006520 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7fa80c08b0b0 con 0x7fa81406bcb0 2024-04-24T23:48:53.469 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-04-24T23:48:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.472+0000 7fa8189b7640 1 -- 172.21.15.16:0/2932006520 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa7f406be30 msgr2=0x7fa7f406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:53.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.472+0000 7fa8189b7640 1 --2- 172.21.15.16:0/2932006520 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa7f406be30 0x7fa7f406e2f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc034000 tx=0x7fa7fc02e000 comp rx=0 tx=0).stop 2024-04-24T23:48:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.473+0000 7fa8189b7640 1 -- 172.21.15.16:0/2932006520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa81406bcb0 msgr2=0x7fa8141f6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.473+0000 7fa8189b7640 1 --2- 172.21.15.16:0/2932006520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa81406bcb0 0x7fa8141f6860 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa80c00c430 tx=0x7fa80c00c8f0 comp rx=0 tx=0).stop 2024-04-24T23:48:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.473+0000 7fa8189b7640 1 -- 172.21.15.16:0/2932006520 shutdown_connections 2024-04-24T23:48:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.473+0000 7fa8189b7640 1 --2- 172.21.15.16:0/2932006520 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa7f406be30 0x7fa7f406e2f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.473+0000 7fa8189b7640 1 --2- 172.21.15.16:0/2932006520 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa81415dfb0 0x7fa8140b5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.473+0000 7fa8189b7640 1 --2- 172.21.15.16:0/2932006520 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa8140d2260 0x7fa8140b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.473+0000 7fa8189b7640 1 --2- 172.21.15.16:0/2932006520 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa81406bcb0 0x7fa8141f6860 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:53.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.473+0000 7fa8189b7640 1 -- 172.21.15.16:0/2932006520 >> 172.21.15.16:0/2932006520 conn(0x7fa8140d3e30 msgr2=0x7fa8140b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:53.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.473+0000 7fa8189b7640 1 -- 172.21.15.16:0/2932006520 shutdown_connections 2024-04-24T23:48:53.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.475+0000 7fa8189b7640 1 -- 172.21.15.16:0/2932006520 wait complete. 2024-04-24T23:48:53.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-24T23:48:53.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.598+0000 7f4d2e514640 1 Processor -- start 2024-04-24T23:48:53.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.598+0000 7f4d2e514640 1 -- start start 2024-04-24T23:48:53.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.598+0000 7f4d2e514640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d280d12d0 0x7f4d280d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:53.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.598+0000 7f4d2e514640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d2815ec30 con 0x7f4d280cd790 2024-04-24T23:48:53.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.598+0000 7f4d2e514640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d2815ef10 con 0x7f4d2815e5c0 2024-04-24T23:48:53.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.598+0000 7f4d2e514640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f4d2815f4d0 con 0x7f4d280d12d0 2024-04-24T23:48:53.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d277fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4d280cd790 0x7f4d280c8580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60754/0 (socket says 172.21.15.16:60754) 2024-04-24T23:48:53.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d277fe640 1 -- 172.21.15.16:0/3315527562 learned_addr learned my addr 172.21.15.16:0/3315527562 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:48:53.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d27fff640 1 --2- 172.21.15.16:0/3315527562 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d280d12d0 0x7f4d280d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:53.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 863904490 0 0) 0x7f4d2815ec30 con 0x7f4d280cd790 2024-04-24T23:48:53.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d10003520 con 0x7f4d280cd790 2024-04-24T23:48:53.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2954260491 0 0) 0x7f4d2815ef10 con 0x7f4d2815e5c0 2024-04-24T23:48:53.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d2815ec30 con 0x7f4d2815e5c0 2024-04-24T23:48:53.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3478556347 0 0) 0x7f4d10003520 con 0x7f4d280cd790 2024-04-24T23:48:53.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d280d12d0 msgr2=0x7f4d280d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:53.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 --2- 172.21.15.16:0/3315527562 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d280d12d0 0x7f4d280d16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:53.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 >> v1:172.21.15.156:6789/0 conn(0x7f4d2815e5c0 legacy=0x7f4d280d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:53.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d2815f1f0 con 0x7f4d280cd790 2024-04-24T23:48:53.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f4d1c001670 con 0x7f4d280cd790 2024-04-24T23:48:53.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 >> v1:172.21.15.16:6789/0 conn(0x7f4d280cd790 legacy=0x7f4d280c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:53.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 --2- 172.21.15.16:0/3315527562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d10003b70 0x7f4d10003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:53.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 --2- 172.21.15.16:0/3315527562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d10004800 0x7f4d10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:53.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 --2- 172.21.15.16:0/3315527562 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d280d12d0 0x7f4d10005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:53.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d2815f4d0 con 0x7f4d10003b70 2024-04-24T23:48:53.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4d1000bb80 con 0x7f4d10004800 2024-04-24T23:48:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f4d1000bd80 con 0x7f4d280d12d0 2024-04-24T23:48:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d27fff640 1 --2- 172.21.15.16:0/3315527562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d10003b70 0x7f4d10003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d277fe640 1 --2- 172.21.15.16:0/3315527562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d10004800 0x7f4d10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.599+0000 7f4d2cab1640 1 --2- 172.21.15.16:0/3315527562 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d280d12d0 0x7f4d10005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.600+0000 7f4d27fff640 1 -- 172.21.15.16:0/3315527562 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d280d12d0 msgr2=0x7f4d10005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.600+0000 7f4d27fff640 1 --2- 172.21.15.16:0/3315527562 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d280d12d0 0x7f4d10005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.600+0000 7f4d27fff640 1 -- 172.21.15.16:0/3315527562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d10004800 msgr2=0x7f4d10004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.600+0000 7f4d27fff640 1 --2- 172.21.15.16:0/3315527562 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d10004800 0x7f4d10004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.600+0000 7f4d27fff640 1 -- 172.21.15.16:0/3315527562 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d10003520 con 0x7f4d10003b70 2024-04-24T23:48:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.600+0000 7f4d27fff640 1 --2- 172.21.15.16:0/3315527562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d10003b70 0x7f4d10003f70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009ac0 tx=0x7f4d10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14c17b6f024e713 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.600+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18036070 con 0x7f4d10003b70 2024-04-24T23:48:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.600+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d18031400 con 0x7f4d10003b70 2024-04-24T23:48:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.600+0000 7f4d26ffd640 1 -- 172.21.15.16:0/3315527562 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d18009690 con 0x7f4d10003b70 2024-04-24T23:48:53.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.601+0000 7f4d2e514640 1 -- 172.21.15.16:0/3315527562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d10003b70 msgr2=0x7f4d10003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.601+0000 7f4d2e514640 1 --2- 172.21.15.16:0/3315527562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d10003b70 0x7f4d10003f70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009ac0 tx=0x7f4d10005830 comp rx=0 tx=0).stop 2024-04-24T23:48:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.601+0000 7f4d2cab1640 1 -- 172.21.15.16:0/3315527562 reap_dead start 2024-04-24T23:48:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.601+0000 7f4d2e514640 1 -- 172.21.15.16:0/3315527562 shutdown_connections 2024-04-24T23:48:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.601+0000 7f4d2e514640 1 -- 172.21.15.16:0/3315527562 >> 172.21.15.16:0/3315527562 conn(0x7f4d28072e70 msgr2=0x7f4d28073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.601+0000 7f4d2e514640 1 -- 172.21.15.16:0/3315527562 shutdown_connections 2024-04-24T23:48:53.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.601+0000 7f4d2e514640 1 -- 172.21.15.16:0/3315527562 wait complete. 2024-04-24T23:48:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.602+0000 7f4d2e514640 1 Processor -- start 2024-04-24T23:48:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.602+0000 7f4d2e514640 1 -- start start 2024-04-24T23:48:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d2e514640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d10003b70 0x7f4d28156460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d2e514640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d281569a0 0x7f4d280c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d2e514640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d281f3c90 0x7f4d280c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d2e514640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d280d29c0 con 0x7f4d281569a0 2024-04-24T23:48:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d2e514640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4d280d2840 con 0x7f4d281f3c90 2024-04-24T23:48:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d2e514640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f4d280d16b0 con 0x7f4d10003b70 2024-04-24T23:48:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d281569a0 0x7f4d280c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d281569a0 0x7f4d280c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43814/0 (socket says 172.21.15.16:43814) 2024-04-24T23:48:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d277fe640 1 -- 172.21.15.16:0/1474634988 learned_addr learned my addr 172.21.15.16:0/1474634988 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:53.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d27fff640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d10003b70 0x7f4d28156460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d2cab1640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d281f3c90 0x7f4d280c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d277fe640 1 -- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d10003b70 msgr2=0x7f4d28156460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d277fe640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d10003b70 0x7f4d28156460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d277fe640 1 -- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d281f3c90 msgr2=0x7f4d280c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d277fe640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d281f3c90 0x7f4d280c9230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d277fe640 1 -- 172.21.15.16:0/1474634988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d28156e60 con 0x7f4d281569a0 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.603+0000 7f4d27fff640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d10003b70 0x7f4d28156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.604+0000 7f4d277fe640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d281569a0 0x7f4d280c8cf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c000f60 tx=0x7f4d1c00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.604+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1474634988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c00c870 con 0x7f4d281569a0 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.604+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1474634988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d1c018070 con 0x7f4d281569a0 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.604+0000 7f4d2e514640 1 -- 172.21.15.16:0/1474634988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d28157140 con 0x7f4d281569a0 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.604+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1474634988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d1c00cc50 con 0x7f4d281569a0 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.604+0000 7f4d2e514640 1 -- 172.21.15.16:0/1474634988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d28157700 con 0x7f4d281569a0 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.605+0000 7f4d2e514640 1 -- 172.21.15.16:0/1474634988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d281520b0 con 0x7f4d281569a0 2024-04-24T23:48:53.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.605+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1474634988 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4d1c01d460 con 0x7f4d281569a0 2024-04-24T23:48:53.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.606+0000 7f4d24ff9640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4d1406be30 0x7f4d1406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:53.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.606+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1474634988 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f4d1c08ab10 con 0x7f4d281569a0 2024-04-24T23:48:53.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.606+0000 7f4d27fff640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4d1406be30 0x7f4d1406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:53.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.607+0000 7f4d27fff640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4d1406be30 0x7f4d1406e2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4d18033000 tx=0x7f4d18008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:53.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.611+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1474634988 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4d281520b0 con 0x7f4d281569a0 2024-04-24T23:48:53.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:53.792+0000 7f4d2e514640 1 -- 172.21.15.16:0/1474634988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f4d280c9f10 con 0x7f4d281569a0 2024-04-24T23:48:53.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:48:53.858 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:48:54.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.475+0000 7f4d24ff9640 1 -- 172.21.15.16:0/1474634988 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) ==== 171+0+0 (secure 0 0 0) 0x7f4d1c0596a0 con 0x7f4d281569a0 2024-04-24T23:48:54.476 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-04-24T23:48:54.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.480+0000 7f4d2e514640 1 -- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4d1406be30 msgr2=0x7f4d1406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:54.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.480+0000 7f4d2e514640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4d1406be30 0x7f4d1406e2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4d18033000 tx=0x7f4d18008030 comp rx=0 tx=0).stop 2024-04-24T23:48:54.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.480+0000 7f4d2e514640 1 -- 172.21.15.16:0/1474634988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d281569a0 msgr2=0x7f4d280c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:54.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.480+0000 7f4d2e514640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d281569a0 0x7f4d280c8cf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4d1c000f60 tx=0x7f4d1c00ec60 comp rx=0 tx=0).stop 2024-04-24T23:48:54.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.480+0000 7f4d2e514640 1 -- 172.21.15.16:0/1474634988 shutdown_connections 2024-04-24T23:48:54.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.480+0000 7f4d2e514640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4d1406be30 0x7f4d1406e2f0 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4d18033000 tx=0x7f4d18008030 comp rx=0 tx=0).stop 2024-04-24T23:48:54.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.480+0000 7f4d2e514640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d281f3c90 0x7f4d280c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:54.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.480+0000 7f4d2e514640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d281569a0 0x7f4d280c8cf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:54.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.480+0000 7f4d2e514640 1 --2- 172.21.15.16:0/1474634988 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d10003b70 0x7f4d28156460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:54.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.480+0000 7f4d2e514640 1 -- 172.21.15.16:0/1474634988 >> 172.21.15.16:0/1474634988 conn(0x7f4d28072e70 msgr2=0x7f4d280cedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:54.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.481+0000 7f4d2e514640 1 -- 172.21.15.16:0/1474634988 shutdown_connections 2024-04-24T23:48:54.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.484+0000 7f4d2e514640 1 -- 172.21.15.16:0/1474634988 wait complete. 2024-04-24T23:48:54.507 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-24T23:48:54.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.615+0000 7fe31c6ba640 1 Processor -- start 2024-04-24T23:48:54.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.615+0000 7fe31c6ba640 1 -- start start 2024-04-24T23:48:54.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.616+0000 7fe31c6ba640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3141456b0 0x7fe3140b3000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.616+0000 7fe31c6ba640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe3140d1b50 0x7fe3140d1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.616+0000 7fe31c6ba640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe31415d140 con 0x7fe3140b5900 2024-04-24T23:48:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.616+0000 7fe31c6ba640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fe3140d24f0 con 0x7fe3140d1b50 2024-04-24T23:48:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.616+0000 7fe31c6ba640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe3140b2520 con 0x7fe3141456b0 2024-04-24T23:48:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.616+0000 7fe31ac57640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe3140b5900 0x7fe3140b5d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60764/0 (socket says 172.21.15.16:60764) 2024-04-24T23:48:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.616+0000 7fe31ac57640 1 -- 172.21.15.16:0/4212673146 learned_addr learned my addr 172.21.15.16:0/4212673146 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:48:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.616+0000 7fe319c55640 1 --2- 172.21.15.16:0/4212673146 >> v2:172.21.15.156:3300/0 conn(0x7fe3140d1b50 0x7fe3140d1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.616+0000 7fe31a456640 1 --2- 172.21.15.16:0/4212673146 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3141456b0 0x7fe3140b3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.616+0000 7fe319454640 1 -- 172.21.15.16:0/4212673146 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 535347874 0 0) 0x7fe31415d140 con 0x7fe3140b5900 2024-04-24T23:48:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.617+0000 7fe319454640 1 -- 172.21.15.16:0/4212673146 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2fc003520 con 0x7fe3140b5900 2024-04-24T23:48:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.617+0000 7fe319c55640 1 -- 172.21.15.16:0/4212673146 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3141456b0 msgr2=0x7fe3140b3000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.617+0000 7fe319c55640 1 --2- 172.21.15.16:0/4212673146 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3141456b0 0x7fe3140b3000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.617+0000 7fe319c55640 1 -- 172.21.15.16:0/4212673146 >> v1:172.21.15.16:6789/0 conn(0x7fe3140b5900 legacy=0x7fe3140b5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:54.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.617+0000 7fe319c55640 1 -- 172.21.15.16:0/4212673146 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3140b2800 con 0x7fe3140d1b50 2024-04-24T23:48:54.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.617+0000 7fe31a456640 1 --2- 172.21.15.16:0/4212673146 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3141456b0 0x7fe3140b3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:54.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.617+0000 7fe319c55640 1 --2- 172.21.15.16:0/4212673146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3140d1b50 0x7fe3140d1fb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe3080097a0 tx=0x7fe30802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e238868051358b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:54.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.618+0000 7fe319454640 1 -- 172.21.15.16:0/4212673146 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308037070 con 0x7fe3140d1b50 2024-04-24T23:48:54.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.618+0000 7fe319454640 1 -- 172.21.15.16:0/4212673146 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe308032440 con 0x7fe3140d1b50 2024-04-24T23:48:54.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.618+0000 7fe319454640 1 -- 172.21.15.16:0/4212673146 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe308032810 con 0x7fe3140d1b50 2024-04-24T23:48:54.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.618+0000 7fe31c6ba640 1 -- 172.21.15.16:0/4212673146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3140d1b50 msgr2=0x7fe3140d1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:54.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.618+0000 7fe31c6ba640 1 --2- 172.21.15.16:0/4212673146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3140d1b50 0x7fe3140d1fb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe3080097a0 tx=0x7fe30802a180 comp rx=0 tx=0).stop 2024-04-24T23:48:54.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.618+0000 7fe31c6ba640 1 -- 172.21.15.16:0/4212673146 shutdown_connections 2024-04-24T23:48:54.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.618+0000 7fe31c6ba640 1 --2- 172.21.15.16:0/4212673146 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3140d1b50 0x7fe3140d1fb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:54.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.618+0000 7fe31c6ba640 1 --2- 172.21.15.16:0/4212673146 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3141456b0 0x7fe3140b3000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:54.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.618+0000 7fe31c6ba640 1 -- 172.21.15.16:0/4212673146 >> 172.21.15.16:0/4212673146 conn(0x7fe31406b2c0 msgr2=0x7fe314072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:54.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.618+0000 7fe31c6ba640 1 -- 172.21.15.16:0/4212673146 shutdown_connections 2024-04-24T23:48:54.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.618+0000 7fe31c6ba640 1 -- 172.21.15.16:0/4212673146 wait complete. 2024-04-24T23:48:54.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.619+0000 7fe31c6ba640 1 Processor -- start 2024-04-24T23:48:54.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.619+0000 7fe31c6ba640 1 -- start start 2024-04-24T23:48:54.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.620+0000 7fe31c6ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3140b5900 0x7fe3140c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:54.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.620+0000 7fe31c6ba640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3140d1b50 0x7fe3140cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:54.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.620+0000 7fe31c6ba640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3141456b0 0x7fe3141eec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:54.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.620+0000 7fe31c6ba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe3140b3540 con 0x7fe3140b5900 2024-04-24T23:48:54.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.620+0000 7fe31c6ba640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe314145a90 con 0x7fe3141456b0 2024-04-24T23:48:54.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.620+0000 7fe31c6ba640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe3140b36c0 con 0x7fe3140d1b50 2024-04-24T23:48:54.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.620+0000 7fe31a456640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3140b5900 0x7fe3140c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:54.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.620+0000 7fe31a456640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3140b5900 0x7fe3140c9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43818/0 (socket says 172.21.15.16:43818) 2024-04-24T23:48:54.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.620+0000 7fe31a456640 1 -- 172.21.15.16:0/3642151810 learned_addr learned my addr 172.21.15.16:0/3642151810 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:54.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.620+0000 7fe319c55640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3140d1b50 0x7fe3140cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:54.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.620+0000 7fe31ac57640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3141456b0 0x7fe3141eec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:54.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.621+0000 7fe31a456640 1 -- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3140d1b50 msgr2=0x7fe3140cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:54.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.621+0000 7fe31a456640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3140d1b50 0x7fe3140cd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:54.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.621+0000 7fe31a456640 1 -- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3141456b0 msgr2=0x7fe3141eec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:54.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.621+0000 7fe31a456640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3141456b0 0x7fe3141eec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:54.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.621+0000 7fe31a456640 1 -- 172.21.15.16:0/3642151810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3140c9e20 con 0x7fe3140b5900 2024-04-24T23:48:54.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.621+0000 7fe31ac57640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3141456b0 0x7fe3141eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:54.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.621+0000 7fe31a456640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3140b5900 0x7fe3140c9720 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe30c00ebe0 tx=0x7fe30c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:54.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.621+0000 7fe312ffd640 1 -- 172.21.15.16:0/3642151810 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c017070 con 0x7fe3140b5900 2024-04-24T23:48:54.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.621+0000 7fe312ffd640 1 -- 172.21.15.16:0/3642151810 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe30c00cc30 con 0x7fe3140b5900 2024-04-24T23:48:54.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.621+0000 7fe31c6ba640 1 -- 172.21.15.16:0/3642151810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3140ca100 con 0x7fe3140b5900 2024-04-24T23:48:54.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.621+0000 7fe312ffd640 1 -- 172.21.15.16:0/3642151810 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c013470 con 0x7fe3140b5900 2024-04-24T23:48:54.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.621+0000 7fe31c6ba640 1 -- 172.21.15.16:0/3642151810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3141f2bd0 con 0x7fe3140b5900 2024-04-24T23:48:54.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.622+0000 7fe31c6ba640 1 -- 172.21.15.16:0/3642151810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3140ca620 con 0x7fe3140b5900 2024-04-24T23:48:54.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.623+0000 7fe312ffd640 1 -- 172.21.15.16:0/3642151810 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe30c01a070 con 0x7fe3140b5900 2024-04-24T23:48:54.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.623+0000 7fe312ffd640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe30006be30 0x7fe30006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:54.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.623+0000 7fe312ffd640 1 -- 172.21.15.16:0/3642151810 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7fe30c08b070 con 0x7fe3140b5900 2024-04-24T23:48:54.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.623+0000 7fe319c55640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe30006be30 0x7fe30006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:54.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.624+0000 7fe319c55640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe30006be30 0x7fe30006e2f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe308034000 tx=0x7fe30802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:54.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.627+0000 7fe312ffd640 1 -- 172.21.15.16:0/3642151810 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe3140ca620 con 0x7fe3140b5900 2024-04-24T23:48:54.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:54.851+0000 7fe31c6ba640 1 -- 172.21.15.16:0/3642151810 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fe3140b6e80 con 0x7fe3140b5900 2024-04-24T23:48:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.483+0000 7fe312ffd640 1 -- 172.21.15.16:0/3642151810 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fe3140b6e80 con 0x7fe3140b5900 2024-04-24T23:48:55.483 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 2 and data pool 3 2024-04-24T23:48:55.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.489+0000 7fe31c6ba640 1 -- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe30006be30 msgr2=0x7fe30006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:55.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.489+0000 7fe31c6ba640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe30006be30 0x7fe30006e2f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe308034000 tx=0x7fe30802e000 comp rx=0 tx=0).stop 2024-04-24T23:48:55.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.489+0000 7fe31c6ba640 1 -- 172.21.15.16:0/3642151810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3140b5900 msgr2=0x7fe3140c9720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:55.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.489+0000 7fe31c6ba640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3140b5900 0x7fe3140c9720 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe30c00ebe0 tx=0x7fe30c00c520 comp rx=0 tx=0).stop 2024-04-24T23:48:55.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.489+0000 7fe31c6ba640 1 -- 172.21.15.16:0/3642151810 shutdown_connections 2024-04-24T23:48:55.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.489+0000 7fe31c6ba640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe30006be30 0x7fe30006e2f0 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe308034000 tx=0x7fe30802e000 comp rx=0 tx=0).stop 2024-04-24T23:48:55.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.489+0000 7fe31c6ba640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3141456b0 0x7fe3141eec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:55.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.489+0000 7fe31c6ba640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3140d1b50 0x7fe3140cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:55.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.489+0000 7fe31c6ba640 1 --2- 172.21.15.16:0/3642151810 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3140b5900 0x7fe3140c9720 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:55.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.489+0000 7fe31c6ba640 1 -- 172.21.15.16:0/3642151810 >> 172.21.15.16:0/3642151810 conn(0x7fe31406b2c0 msgr2=0x7fe314070370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:55.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.492+0000 7fe31c6ba640 1 -- 172.21.15.16:0/3642151810 shutdown_connections 2024-04-24T23:48:55.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.494+0000 7fe31c6ba640 1 -- 172.21.15.16:0/3642151810 wait complete. 2024-04-24T23:48:55.517 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-24T23:48:55.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-24T23:48:55.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.629+0000 7f7102110640 1 Processor -- start 2024-04-24T23:48:55.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.629+0000 7f7102110640 1 -- start start 2024-04-24T23:48:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.629+0000 7f7102110640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f70fc15dfb0 0x7f70fc0cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.629+0000 7f7102110640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70fc06c110 0x7f70fc162250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.629+0000 7f7102110640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f70fc0d1aa0 con 0x7f70fc06bcb0 2024-04-24T23:48:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.629+0000 7f7102110640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f70fc0d1dd0 con 0x7f70fc06c110 2024-04-24T23:48:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.629+0000 7f7102110640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f70fc06b470 con 0x7f70fc15dfb0 2024-04-24T23:48:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.630+0000 7f70fbfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70fc06c110 0x7f70fc162250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.630+0000 7f70faffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f70fc06bcb0 0x7f70fc161b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60772/0 (socket says 172.21.15.16:60772) 2024-04-24T23:48:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.630+0000 7f70faffd640 1 -- 172.21.15.16:0/3915057023 learned_addr learned my addr 172.21.15.16:0/3915057023 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:48:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.630+0000 7f70fb7fe640 1 --2- 172.21.15.16:0/3915057023 >> v2:172.21.15.156:3300/0 conn(0x7f70fc15dfb0 0x7f70fc0cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.630+0000 7f70fbfff640 1 -- 172.21.15.16:0/3915057023 >> v2:172.21.15.156:3300/0 conn(0x7f70fc15dfb0 msgr2=0x7f70fc0cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:55.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.630+0000 7f70fbfff640 1 --2- 172.21.15.16:0/3915057023 >> v2:172.21.15.156:3300/0 conn(0x7f70fc15dfb0 0x7f70fc0cfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.630+0000 7f70fbfff640 1 -- 172.21.15.16:0/3915057023 >> v1:172.21.15.16:6789/0 conn(0x7f70fc06bcb0 legacy=0x7f70fc161b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:55.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.630+0000 7f70fbfff640 1 -- 172.21.15.16:0/3915057023 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70fc06b730 con 0x7f70fc06c110 2024-04-24T23:48:55.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.631+0000 7f70fbfff640 1 --2- 172.21.15.16:0/3915057023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70fc06c110 0x7f70fc162250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f70fc0b69d0 tx=0x7f70f402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd35ace0693800b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:55.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.631+0000 7f70fa7fc640 1 -- 172.21.15.16:0/3915057023 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f4007740 con 0x7f70fc06c110 2024-04-24T23:48:55.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.631+0000 7f70fa7fc640 1 -- 172.21.15.16:0/3915057023 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70f40078e0 con 0x7f70fc06c110 2024-04-24T23:48:55.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.631+0000 7f70fa7fc640 1 -- 172.21.15.16:0/3915057023 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70f4007cb0 con 0x7f70fc06c110 2024-04-24T23:48:55.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.631+0000 7f7102110640 1 -- 172.21.15.16:0/3915057023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70fc06c110 msgr2=0x7f70fc162250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:55.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.631+0000 7f7102110640 1 --2- 172.21.15.16:0/3915057023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70fc06c110 0x7f70fc162250 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f70fc0b69d0 tx=0x7f70f402a680 comp rx=0 tx=0).stop 2024-04-24T23:48:55.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.631+0000 7f7102110640 1 -- 172.21.15.16:0/3915057023 shutdown_connections 2024-04-24T23:48:55.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.631+0000 7f7102110640 1 --2- 172.21.15.16:0/3915057023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70fc06c110 0x7f70fc162250 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:55.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.631+0000 7f7102110640 1 --2- 172.21.15.16:0/3915057023 >> v2:172.21.15.156:3300/0 conn(0x7f70fc15dfb0 0x7f70fc0cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:55.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.631+0000 7f7102110640 1 -- 172.21.15.16:0/3915057023 >> 172.21.15.16:0/3915057023 conn(0x7f70fc0d3e30 msgr2=0x7f70fc0d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:55.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.631+0000 7f7102110640 1 -- 172.21.15.16:0/3915057023 shutdown_connections 2024-04-24T23:48:55.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.631+0000 7f7102110640 1 -- 172.21.15.16:0/3915057023 wait complete. 2024-04-24T23:48:55.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.632+0000 7f7102110640 1 Processor -- start 2024-04-24T23:48:55.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.633+0000 7f7102110640 1 -- start start 2024-04-24T23:48:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.633+0000 7f7102110640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70fc06bcb0 0x7f70fc0c9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.633+0000 7f7102110640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70fc06c110 0x7f70fc14b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:55.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.633+0000 7f7102110640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70fc15dfb0 0x7f70fc14b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.633+0000 7f7102110640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f70fc06c560 con 0x7f70fc06bcb0 2024-04-24T23:48:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.633+0000 7f7102110640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f70fc06ba10 con 0x7f70fc15dfb0 2024-04-24T23:48:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.633+0000 7f7102110640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f70fc0b6f60 con 0x7f70fc06c110 2024-04-24T23:48:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.633+0000 7f70fb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70fc06bcb0 0x7f70fc0c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f70fb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70fc06bcb0 0x7f70fc0c9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43848/0 (socket says 172.21.15.16:43848) 2024-04-24T23:48:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f70fb7fe640 1 -- 172.21.15.16:0/2166868227 learned_addr learned my addr 172.21.15.16:0/2166868227 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f70fbfff640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70fc15dfb0 0x7f70fc14b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f70faffd640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70fc06c110 0x7f70fc14b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f70fb7fe640 1 -- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70fc06c110 msgr2=0x7f70fc14b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f70fb7fe640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70fc06c110 0x7f70fc14b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f70fb7fe640 1 -- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70fc15dfb0 msgr2=0x7f70fc14b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f70fb7fe640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70fc15dfb0 0x7f70fc14b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f70fb7fe640 1 -- 172.21.15.16:0/2166868227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70fc0c80a0 con 0x7f70fc06bcb0 2024-04-24T23:48:55.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f70faffd640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70fc06c110 0x7f70fc14b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:48:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f70fb7fe640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70fc06bcb0 0x7f70fc0c9100 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f70ec00e9b0 tx=0x7f70ec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f70e3fff640 1 -- 172.21.15.16:0/2166868227 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ec017070 con 0x7f70fc06bcb0 2024-04-24T23:48:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f70e3fff640 1 -- 172.21.15.16:0/2166868227 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70ec00c9a0 con 0x7f70fc06bcb0 2024-04-24T23:48:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f7102110640 1 -- 172.21.15.16:0/2166868227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70fc0c8640 con 0x7f70fc06bcb0 2024-04-24T23:48:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f70e3fff640 1 -- 172.21.15.16:0/2166868227 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70ec00cd90 con 0x7f70fc06bcb0 2024-04-24T23:48:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.634+0000 7f7102110640 1 -- 172.21.15.16:0/2166868227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70fc0c8c00 con 0x7f70fc06bcb0 2024-04-24T23:48:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.635+0000 7f7102110640 1 -- 172.21.15.16:0/2166868227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70fc06d3c0 con 0x7f70fc06bcb0 2024-04-24T23:48:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.636+0000 7f70e3fff640 1 -- 172.21.15.16:0/2166868227 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f70ec01a070 con 0x7f70fc06bcb0 2024-04-24T23:48:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.636+0000 7f70e3fff640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f70e806be30 0x7f70e806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:55.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.636+0000 7f70e3fff640 1 -- 172.21.15.16:0/2166868227 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f70ec08ad10 con 0x7f70fc06bcb0 2024-04-24T23:48:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.636+0000 7f70faffd640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f70e806be30 0x7f70e806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:55.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.637+0000 7f70faffd640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f70e806be30 0x7f70e806e2f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f70fc0c9640 tx=0x7f70f0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:55.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.640+0000 7f70e3fff640 1 -- 172.21.15.16:0/2166868227 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f70fc06d3c0 con 0x7f70fc06bcb0 2024-04-24T23:48:55.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:55.836+0000 7f7102110640 1 -- 172.21.15.16:0/2166868227 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f70fc0cecb0 con 0x7f70fc06bcb0 2024-04-24T23:48:56.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.489+0000 7f70e3fff640 1 -- 172.21.15.16:0/2166868227 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) ==== 157+0+0 (secure 0 0 0) 0x7f70ec059100 con 0x7f70fc06bcb0 2024-04-24T23:48:56.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.493+0000 7f7102110640 1 -- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f70e806be30 msgr2=0x7f70e806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:56.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.493+0000 7f7102110640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f70e806be30 0x7f70e806e2f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f70fc0c9640 tx=0x7f70f0009040 comp rx=0 tx=0).stop 2024-04-24T23:48:56.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.493+0000 7f7102110640 1 -- 172.21.15.16:0/2166868227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70fc06bcb0 msgr2=0x7f70fc0c9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:56.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.493+0000 7f7102110640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70fc06bcb0 0x7f70fc0c9100 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f70ec00e9b0 tx=0x7f70ec00ee70 comp rx=0 tx=0).stop 2024-04-24T23:48:56.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.493+0000 7f7102110640 1 -- 172.21.15.16:0/2166868227 shutdown_connections 2024-04-24T23:48:56.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.493+0000 7f7102110640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f70e806be30 0x7f70e806e2f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:56.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.493+0000 7f7102110640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70fc15dfb0 0x7f70fc14b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:56.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.493+0000 7f7102110640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70fc06c110 0x7f70fc14b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:56.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.493+0000 7f7102110640 1 --2- 172.21.15.16:0/2166868227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70fc06bcb0 0x7f70fc0c9100 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:56.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.493+0000 7f7102110640 1 -- 172.21.15.16:0/2166868227 >> 172.21.15.16:0/2166868227 conn(0x7f70fc0d3e30 msgr2=0x7f70fc15be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:56.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.493+0000 7f7102110640 1 -- 172.21.15.16:0/2166868227 shutdown_connections 2024-04-24T23:48:56.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.493+0000 7f7102110640 1 -- 172.21.15.16:0/2166868227 wait complete. 2024-04-24T23:48:56.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-24T23:48:56.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.622+0000 7fd29901c640 1 Processor -- start 2024-04-24T23:48:56.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.622+0000 7fd29901c640 1 -- start start 2024-04-24T23:48:56.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.622+0000 7fd29901c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd2941478d0 0x7fd2940d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:56.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.622+0000 7fd29901c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd2940d2090 0x7fd2940b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:56.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.622+0000 7fd29901c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2940b2a10 con 0x7fd2940b21d0 2024-04-24T23:48:56.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.622+0000 7fd29901c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd2940d24f0 con 0x7fd2941478d0 2024-04-24T23:48:56.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.622+0000 7fd29901c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fd2940b3620 con 0x7fd2940d2090 2024-04-24T23:48:56.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.623+0000 7fd292d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd2941478d0 0x7fd2940d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:56.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.623+0000 7fd292d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd2941478d0 0x7fd2940d1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43852/0 (socket says 172.21.15.16:43852) 2024-04-24T23:48:56.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.623+0000 7fd292d9d640 1 -- 172.21.15.16:0/4065678846 learned_addr learned my addr 172.21.15.16:0/4065678846 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:56.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.623+0000 7fd28bfff640 1 --2- 172.21.15.16:0/4065678846 >> v2:172.21.15.156:3300/0 conn(0x7fd2940d2090 0x7fd2940b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:56.623 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.623+0000 7fd292d9d640 1 -- 172.21.15.16:0/4065678846 >> v2:172.21.15.156:3300/0 conn(0x7fd2940d2090 msgr2=0x7fd2940b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:56.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.623+0000 7fd292d9d640 1 --2- 172.21.15.16:0/4065678846 >> v2:172.21.15.156:3300/0 conn(0x7fd2940d2090 0x7fd2940b1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:56.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.623+0000 7fd292d9d640 1 -- 172.21.15.16:0/4065678846 >> v1:172.21.15.156:6789/0 conn(0x7fd2940b21d0 legacy=0x7fd2940d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:56.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.623+0000 7fd292d9d640 1 -- 172.21.15.16:0/4065678846 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2940b2cf0 con 0x7fd2941478d0 2024-04-24T23:48:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.624+0000 7fd292d9d640 1 --2- 172.21.15.16:0/4065678846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2941478d0 0x7fd2940d1b50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd294072660 tx=0x7fd28402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60503af965602203 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.624+0000 7fd29259c640 1 -- 172.21.15.16:0/4065678846 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd284037070 con 0x7fd2941478d0 2024-04-24T23:48:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.624+0000 7fd29259c640 1 -- 172.21.15.16:0/4065678846 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd284032440 con 0x7fd2941478d0 2024-04-24T23:48:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.624+0000 7fd29259c640 1 -- 172.21.15.16:0/4065678846 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd284032810 con 0x7fd2941478d0 2024-04-24T23:48:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.624+0000 7fd29901c640 1 -- 172.21.15.16:0/4065678846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2941478d0 msgr2=0x7fd2940d1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.624+0000 7fd29901c640 1 --2- 172.21.15.16:0/4065678846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2941478d0 0x7fd2940d1b50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd294072660 tx=0x7fd28402a230 comp rx=0 tx=0).stop 2024-04-24T23:48:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.624+0000 7fd29901c640 1 -- 172.21.15.16:0/4065678846 shutdown_connections 2024-04-24T23:48:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.624+0000 7fd29901c640 1 --2- 172.21.15.16:0/4065678846 >> v2:172.21.15.156:3300/0 conn(0x7fd2940d2090 0x7fd2940b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.624+0000 7fd29901c640 1 --2- 172.21.15.16:0/4065678846 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2941478d0 0x7fd2940d1b50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.624+0000 7fd29901c640 1 -- 172.21.15.16:0/4065678846 >> 172.21.15.16:0/4065678846 conn(0x7fd29406b2c0 msgr2=0x7fd294072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:56.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.624+0000 7fd29901c640 1 -- 172.21.15.16:0/4065678846 shutdown_connections 2024-04-24T23:48:56.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.625+0000 7fd29901c640 1 -- 172.21.15.16:0/4065678846 wait complete. 2024-04-24T23:48:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.626+0000 7fd29901c640 1 Processor -- start 2024-04-24T23:48:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.626+0000 7fd29901c640 1 -- start start 2024-04-24T23:48:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29901c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd2940b21d0 0x7fd294136b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29901c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2940d2090 0x7fd2941e64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29901c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2941478d0 0x7fd2941e6a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29901c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd2940b2640 con 0x7fd2941478d0 2024-04-24T23:48:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29901c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd2940b32b0 con 0x7fd2940d2090 2024-04-24T23:48:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29901c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fd2940b27c0 con 0x7fd2940b21d0 2024-04-24T23:48:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2941478d0 0x7fd2941e6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2941478d0 0x7fd2941e6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43868/0 (socket says 172.21.15.16:43868) 2024-04-24T23:48:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29359e640 1 -- 172.21.15.16:0/275042349 learned_addr learned my addr 172.21.15.16:0/275042349 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd28bfff640 1 --2- 172.21.15.16:0/275042349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2940d2090 0x7fd2941e64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd292d9d640 1 --2- 172.21.15.16:0/275042349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd2940b21d0 0x7fd294136b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29359e640 1 -- 172.21.15.16:0/275042349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd2940b21d0 msgr2=0x7fd294136b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29359e640 1 --2- 172.21.15.16:0/275042349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd2940b21d0 0x7fd294136b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29359e640 1 -- 172.21.15.16:0/275042349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2940d2090 msgr2=0x7fd2941e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29359e640 1 --2- 172.21.15.16:0/275042349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2940d2090 0x7fd2941e64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29359e640 1 -- 172.21.15.16:0/275042349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd294137050 con 0x7fd2941478d0 2024-04-24T23:48:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd29359e640 1 --2- 172.21.15.16:0/275042349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2941478d0 0x7fd2941e6a20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00b970 tx=0x7fd28c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.627+0000 7fd28b7fe640 1 -- 172.21.15.16:0/275042349 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28c019070 con 0x7fd2941478d0 2024-04-24T23:48:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.628+0000 7fd28b7fe640 1 -- 172.21.15.16:0/275042349 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd28c00cb40 con 0x7fd2941478d0 2024-04-24T23:48:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.628+0000 7fd28b7fe640 1 -- 172.21.15.16:0/275042349 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd28c014440 con 0x7fd2941478d0 2024-04-24T23:48:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.628+0000 7fd29901c640 1 -- 172.21.15.16:0/275042349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2940cdc60 con 0x7fd2941478d0 2024-04-24T23:48:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.628+0000 7fd29901c640 1 -- 172.21.15.16:0/275042349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2940ce220 con 0x7fd2941478d0 2024-04-24T23:48:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.628+0000 7fd29901c640 1 -- 172.21.15.16:0/275042349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd294068f90 con 0x7fd2941478d0 2024-04-24T23:48:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.629+0000 7fd28b7fe640 1 -- 172.21.15.16:0/275042349 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd28c01c070 con 0x7fd2941478d0 2024-04-24T23:48:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.629+0000 7fd28b7fe640 1 --2- 172.21.15.16:0/275042349 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd27406be30 0x7fd27406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.630+0000 7fd28b7fe640 1 -- 172.21.15.16:0/275042349 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 6007+0+0 (secure 0 0 0) 0x7fd28c08ab50 con 0x7fd2941478d0 2024-04-24T23:48:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.630+0000 7fd292d9d640 1 --2- 172.21.15.16:0/275042349 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd27406be30 0x7fd27406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.630+0000 7fd292d9d640 1 --2- 172.21.15.16:0/275042349 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd27406be30 0x7fd27406e2f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd284034000 tx=0x7fd28402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:56.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.634+0000 7fd28b7fe640 1 -- 172.21.15.16:0/275042349 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd294137050 con 0x7fd2941478d0 2024-04-24T23:48:56.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:56.819+0000 7fd29901c640 1 -- 172.21.15.16:0/275042349 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd2940b1ab0 con 0x7fd2941478d0 2024-04-24T23:48:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.499+0000 7fd28b7fe640 1 -- 172.21.15.16:0/275042349 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) ==== 228+0+0 (secure 0 0 0) 0x7fd28c058d80 con 0x7fd2941478d0 2024-04-24T23:48:58.499 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data_ec' created 2024-04-24T23:48:58.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.503+0000 7fd29901c640 1 -- 172.21.15.16:0/275042349 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd27406be30 msgr2=0x7fd27406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:58.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.503+0000 7fd29901c640 1 --2- 172.21.15.16:0/275042349 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd27406be30 0x7fd27406e2f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd284034000 tx=0x7fd28402e000 comp rx=0 tx=0).stop 2024-04-24T23:48:58.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.503+0000 7fd29901c640 1 -- 172.21.15.16:0/275042349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2941478d0 msgr2=0x7fd2941e6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:58.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.503+0000 7fd29901c640 1 --2- 172.21.15.16:0/275042349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2941478d0 0x7fd2941e6a20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00b970 tx=0x7fd28c00be30 comp rx=0 tx=0).stop 2024-04-24T23:48:58.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.503+0000 7fd29901c640 1 -- 172.21.15.16:0/275042349 shutdown_connections 2024-04-24T23:48:58.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.503+0000 7fd29901c640 1 --2- 172.21.15.16:0/275042349 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd27406be30 0x7fd27406e2f0 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd284034000 tx=0x7fd28402e000 comp rx=0 tx=0).stop 2024-04-24T23:48:58.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.503+0000 7fd29901c640 1 --2- 172.21.15.16:0/275042349 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2941478d0 0x7fd2941e6a20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.503+0000 7fd29901c640 1 --2- 172.21.15.16:0/275042349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd2940d2090 0x7fd2941e64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.503+0000 7fd29901c640 1 --2- 172.21.15.16:0/275042349 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd2940b21d0 0x7fd294136b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.503+0000 7fd29901c640 1 -- 172.21.15.16:0/275042349 >> 172.21.15.16:0/275042349 conn(0x7fd29406b2c0 msgr2=0x7fd294145e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.504+0000 7fd29901c640 1 -- 172.21.15.16:0/275042349 shutdown_connections 2024-04-24T23:48:58.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.504+0000 7fd29901c640 1 -- 172.21.15.16:0/275042349 wait complete. 2024-04-24T23:48:58.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-24T23:48:58.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.634+0000 7f3958407640 1 Processor -- start 2024-04-24T23:48:58.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.634+0000 7f3958407640 1 -- start start 2024-04-24T23:48:58.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f3958407640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f395006e470 0x7f39500ce440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:58.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f3958407640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f395006e8d0 0x7f39500ce980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:58.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f3958407640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3950073390 con 0x7f39501458a0 2024-04-24T23:48:58.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f3958407640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f39500ceec0 con 0x7f395006e8d0 2024-04-24T23:48:58.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f3958407640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f3950072160 con 0x7f395006e470 2024-04-24T23:48:58.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f39569a4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f395006e8d0 0x7f39500ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:58.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f39561a3640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f39501458a0 0x7f3950072c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44456/0 (socket says 172.21.15.16:44456) 2024-04-24T23:48:58.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f39569a4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f395006e8d0 0x7f39500ce980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43340/0 (socket says 172.21.15.16:43340) 2024-04-24T23:48:58.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f39561a3640 1 -- 172.21.15.16:0/899581865 learned_addr learned my addr 172.21.15.16:0/899581865 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:48:58.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f39559a2640 1 --2- 172.21.15.16:0/899581865 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f395006e470 0x7f39500ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:58.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f39569a4640 1 -- 172.21.15.16:0/899581865 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f395006e470 msgr2=0x7f39500ce440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:58.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f39569a4640 1 --2- 172.21.15.16:0/899581865 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f395006e470 0x7f39500ce440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:58.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f39569a4640 1 -- 172.21.15.16:0/899581865 >> v1:172.21.15.16:6789/0 conn(0x7f39501458a0 legacy=0x7f3950072c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:58.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f39551a1640 1 -- 172.21.15.16:0/899581865 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4220579184 0 0) 0x7f394c001110 con 0x7f39501458a0 2024-04-24T23:48:58.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f39569a4640 1 -- 172.21.15.16:0/899581865 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395006d0f0 con 0x7f395006e8d0 2024-04-24T23:48:58.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.635+0000 7f39559a2640 1 --2- 172.21.15.16:0/899581865 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f395006e470 0x7f39500ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:48:58.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.636+0000 7f39569a4640 1 --2- 172.21.15.16:0/899581865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f395006e8d0 0x7f39500ce980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3948009d90 tx=0x7f394802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d8674d856aafd35 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:58.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.636+0000 7f39551a1640 1 -- 172.21.15.16:0/899581865 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3948031600 con 0x7f395006e8d0 2024-04-24T23:48:58.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.636+0000 7f39551a1640 1 -- 172.21.15.16:0/899581865 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39480317a0 con 0x7f395006e8d0 2024-04-24T23:48:58.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.636+0000 7f39551a1640 1 -- 172.21.15.16:0/899581865 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3948031b70 con 0x7f395006e8d0 2024-04-24T23:48:58.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.637+0000 7f3958407640 1 -- 172.21.15.16:0/899581865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f395006e8d0 msgr2=0x7f39500ce980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:58.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.637+0000 7f3958407640 1 --2- 172.21.15.16:0/899581865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f395006e8d0 0x7f39500ce980 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3948009d90 tx=0x7f394802a5d0 comp rx=0 tx=0).stop 2024-04-24T23:48:58.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.637+0000 7f3958407640 1 -- 172.21.15.16:0/899581865 shutdown_connections 2024-04-24T23:48:58.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.637+0000 7f3958407640 1 --2- 172.21.15.16:0/899581865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f395006e8d0 0x7f39500ce980 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:58.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.637+0000 7f3958407640 1 --2- 172.21.15.16:0/899581865 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f395006e470 0x7f39500ce440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:58.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.637+0000 7f3958407640 1 -- 172.21.15.16:0/899581865 >> 172.21.15.16:0/899581865 conn(0x7f395006b0e0 msgr2=0x7f395006b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:58.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.637+0000 7f3958407640 1 -- 172.21.15.16:0/899581865 shutdown_connections 2024-04-24T23:48:58.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.637+0000 7f3958407640 1 -- 172.21.15.16:0/899581865 wait complete. 2024-04-24T23:48:58.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.638+0000 7f3958407640 1 Processor -- start 2024-04-24T23:48:58.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.638+0000 7f3958407640 1 -- start start 2024-04-24T23:48:58.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f3958407640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f395006e470 0x7f395013c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:58.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f3958407640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f395006e8d0 0x7f395013edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:58.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f3958407640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f39501458a0 0x7f39500aa020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:58.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f3958407640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f395006d990 con 0x7f395006e470 2024-04-24T23:48:58.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f3958407640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f395006d6b0 con 0x7f395006e8d0 2024-04-24T23:48:58.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f3958407640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f395006dc70 con 0x7f39501458a0 2024-04-24T23:48:58.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f39561a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f395006e470 0x7f395013c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:58.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f39561a3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f395006e470 0x7f395013c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43352/0 (socket says 172.21.15.16:43352) 2024-04-24T23:48:58.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f39561a3640 1 -- 172.21.15.16:0/1534301032 learned_addr learned my addr 172.21.15.16:0/1534301032 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:58.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f39561a3640 1 -- 172.21.15.16:0/1534301032 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f39501458a0 msgr2=0x7f39500aa020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:58.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f39561a3640 1 --2- 172.21.15.16:0/1534301032 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f39501458a0 0x7f39500aa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:58.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f39561a3640 1 -- 172.21.15.16:0/1534301032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f395006e8d0 msgr2=0x7f395013edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:58.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f39561a3640 1 --2- 172.21.15.16:0/1534301032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f395006e8d0 0x7f395013edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:58.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f39561a3640 1 -- 172.21.15.16:0/1534301032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f395013f6e0 con 0x7f395006e470 2024-04-24T23:48:58.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f39561a3640 1 --2- 172.21.15.16:0/1534301032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f395006e470 0x7f395013c490 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f394c00c460 tx=0x7f394c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:58.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.639+0000 7f3946ffd640 1 -- 172.21.15.16:0/1534301032 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c018070 con 0x7f395006e470 2024-04-24T23:48:58.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.640+0000 7f3946ffd640 1 -- 172.21.15.16:0/1534301032 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f394c007500 con 0x7f395006e470 2024-04-24T23:48:58.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.640+0000 7f3958407640 1 -- 172.21.15.16:0/1534301032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f395013f9c0 con 0x7f395006e470 2024-04-24T23:48:58.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.640+0000 7f3946ffd640 1 -- 172.21.15.16:0/1534301032 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f394c004a20 con 0x7f395006e470 2024-04-24T23:48:58.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.640+0000 7f3958407640 1 -- 172.21.15.16:0/1534301032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f395013ff80 con 0x7f395006e470 2024-04-24T23:48:58.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.640+0000 7f3958407640 1 -- 172.21.15.16:0/1534301032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39500cf7e0 con 0x7f395006e470 2024-04-24T23:48:58.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.641+0000 7f3946ffd640 1 -- 172.21.15.16:0/1534301032 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f394c01b070 con 0x7f395006e470 2024-04-24T23:48:58.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.641+0000 7f3946ffd640 1 --2- 172.21.15.16:0/1534301032 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f393406be30 0x7f393406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:58.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.642+0000 7f3946ffd640 1 -- 172.21.15.16:0/1534301032 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 6500+0+0 (secure 0 0 0) 0x7f394c08a3d0 con 0x7f395006e470 2024-04-24T23:48:58.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.642+0000 7f39559a2640 1 --2- 172.21.15.16:0/1534301032 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f393406be30 0x7f393406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:58.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.642+0000 7f39559a2640 1 --2- 172.21.15.16:0/1534301032 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f393406be30 0x7f393406e2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3940006fd0 tx=0x7f3940008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:58.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.646+0000 7f3946ffd640 1 -- 172.21.15.16:0/1534301032 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f39500cf7e0 con 0x7f395006e470 2024-04-24T23:48:58.827 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:58.827+0000 7f3958407640 1 -- 172.21.15.16:0/1534301032 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f3950068f90 con 0x7f395006e470 2024-04-24T23:48:59.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.504+0000 7f3946ffd640 1 -- 172.21.15.16:0/1534301032 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) ==== 169+0+0 (secure 0 0 0) 0x7f3950073390 con 0x7f395006e470 2024-04-24T23:48:59.503 INFO:teuthology.orchestra.run.smithi016.stderr:set pool 4 allow_ec_overwrites to true 2024-04-24T23:48:59.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.507+0000 7f3958407640 1 -- 172.21.15.16:0/1534301032 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f393406be30 msgr2=0x7f393406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:59.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.507+0000 7f3958407640 1 --2- 172.21.15.16:0/1534301032 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f393406be30 0x7f393406e2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3940006fd0 tx=0x7f3940008040 comp rx=0 tx=0).stop 2024-04-24T23:48:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.507+0000 7f3958407640 1 -- 172.21.15.16:0/1534301032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f395006e470 msgr2=0x7f395013c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.507+0000 7f3958407640 1 --2- 172.21.15.16:0/1534301032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f395006e470 0x7f395013c490 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f394c00c460 tx=0x7f394c00c920 comp rx=0 tx=0).stop 2024-04-24T23:48:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.507+0000 7f3958407640 1 -- 172.21.15.16:0/1534301032 shutdown_connections 2024-04-24T23:48:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.508+0000 7f3958407640 1 --2- 172.21.15.16:0/1534301032 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f393406be30 0x7f393406e2f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.508+0000 7f3958407640 1 --2- 172.21.15.16:0/1534301032 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f39501458a0 0x7f39500aa020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.508+0000 7f3958407640 1 --2- 172.21.15.16:0/1534301032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f395006e8d0 0x7f395013edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.508+0000 7f3958407640 1 --2- 172.21.15.16:0/1534301032 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f395006e470 0x7f395013c490 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.508+0000 7f3958407640 1 -- 172.21.15.16:0/1534301032 >> 172.21.15.16:0/1534301032 conn(0x7f395006b0e0 msgr2=0x7f3950072780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.508+0000 7f3958407640 1 -- 172.21.15.16:0/1534301032 shutdown_connections 2024-04-24T23:48:59.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.508+0000 7f3958407640 1 -- 172.21.15.16:0/1534301032 wait complete. 2024-04-24T23:48:59.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-24T23:48:59.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.651+0000 7f17f53dc640 1 Processor -- start 2024-04-24T23:48:59.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.651+0000 7f17f53dc640 1 -- start start 2024-04-24T23:48:59.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.652+0000 7f17f53dc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17f01478d0 0x7f17f00d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.652+0000 7f17f53dc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f17f00b23a0 0x7f17f00b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.652+0000 7f17f53dc640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f00b2a10 con 0x7f17f00d2090 2024-04-24T23:48:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.652+0000 7f17f53dc640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f17f00d24f0 con 0x7f17f00b23a0 2024-04-24T23:48:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.652+0000 7f17f53dc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f17f00b3620 con 0x7f17f01478d0 2024-04-24T23:48:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.652+0000 7f17ef7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f17f00b23a0 0x7f17f00b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.652+0000 7f17ee7fc640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f17f00d2090 0x7f17f00b1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:52524/0 (socket says 172.21.15.16:52524) 2024-04-24T23:48:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.652+0000 7f17eeffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17f01478d0 0x7f17f00d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.652+0000 7f17ef7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f17f00b23a0 0x7f17f00b4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:35222/0 (socket says 172.21.15.16:35222) 2024-04-24T23:48:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.652+0000 7f17ee7fc640 1 -- 172.21.15.16:0/1420825361 learned_addr learned my addr 172.21.15.16:0/1420825361 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:48:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.652+0000 7f17ef7fe640 1 -- 172.21.15.16:0/1420825361 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17f01478d0 msgr2=0x7f17f00d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:48:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.652+0000 7f17ef7fe640 1 --2- 172.21.15.16:0/1420825361 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17f01478d0 0x7f17f00d1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.652+0000 7f17ef7fe640 1 -- 172.21.15.16:0/1420825361 >> v1:172.21.15.156:6789/0 conn(0x7f17f00d2090 legacy=0x7f17f00b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.653+0000 7f17ef7fe640 1 -- 172.21.15.16:0/1420825361 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f00b2cf0 con 0x7f17f00b23a0 2024-04-24T23:48:59.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.653+0000 7f17eeffd640 1 --2- 172.21.15.16:0/1420825361 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17f01478d0 0x7f17f00d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:48:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.653+0000 7f17ef7fe640 1 --2- 172.21.15.16:0/1420825361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17f00b23a0 0x7f17f00b4e10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f17e80097a0 tx=0x7f17e802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c142a725f8cfd810 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.653+0000 7f17edffb640 1 -- 172.21.15.16:0/1420825361 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8037070 con 0x7f17f00b23a0 2024-04-24T23:48:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.653+0000 7f17edffb640 1 -- 172.21.15.16:0/1420825361 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17e8032440 con 0x7f17f00b23a0 2024-04-24T23:48:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.653+0000 7f17edffb640 1 -- 172.21.15.16:0/1420825361 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8032810 con 0x7f17f00b23a0 2024-04-24T23:48:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.654+0000 7f17f53dc640 1 -- 172.21.15.16:0/1420825361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17f00b23a0 msgr2=0x7f17f00b4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.654+0000 7f17f53dc640 1 --2- 172.21.15.16:0/1420825361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17f00b23a0 0x7f17f00b4e10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f17e80097a0 tx=0x7f17e802a180 comp rx=0 tx=0).stop 2024-04-24T23:48:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.654+0000 7f17f53dc640 1 -- 172.21.15.16:0/1420825361 shutdown_connections 2024-04-24T23:48:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.654+0000 7f17f53dc640 1 --2- 172.21.15.16:0/1420825361 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17f00b23a0 0x7f17f00b4e10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:59.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.654+0000 7f17f53dc640 1 --2- 172.21.15.16:0/1420825361 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17f01478d0 0x7f17f00d1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:59.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.654+0000 7f17f53dc640 1 -- 172.21.15.16:0/1420825361 >> 172.21.15.16:0/1420825361 conn(0x7f17f006b2c0 msgr2=0x7f17f0072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:48:59.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.654+0000 7f17f53dc640 1 -- 172.21.15.16:0/1420825361 shutdown_connections 2024-04-24T23:48:59.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.654+0000 7f17f53dc640 1 -- 172.21.15.16:0/1420825361 wait complete. 2024-04-24T23:48:59.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.655+0000 7f17f53dc640 1 Processor -- start 2024-04-24T23:48:59.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.655+0000 7f17f53dc640 1 -- start start 2024-04-24T23:48:59.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.656+0000 7f17f53dc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17f00b23a0 0x7f17f015d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:59.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.656+0000 7f17f53dc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17f00d2090 0x7f17f015e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:59.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.656+0000 7f17f53dc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17f01478d0 0x7f17f015e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:59.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.656+0000 7f17f53dc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f17f00b5350 con 0x7f17f01478d0 2024-04-24T23:48:59.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.656+0000 7f17f53dc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f17f00b32b0 con 0x7f17f00b23a0 2024-04-24T23:48:59.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.656+0000 7f17f53dc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f17f00b54d0 con 0x7f17f00d2090 2024-04-24T23:48:59.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.656+0000 7f17ef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17f01478d0 0x7f17f015e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:59.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.656+0000 7f17ef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17f01478d0 0x7f17f015e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43358/0 (socket says 172.21.15.16:43358) 2024-04-24T23:48:59.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.656+0000 7f17ef7fe640 1 -- 172.21.15.16:0/793754741 learned_addr learned my addr 172.21.15.16:0/793754741 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.656+0000 7f17eeffd640 1 --2- 172.21.15.16:0/793754741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17f00b23a0 0x7f17f015d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.656+0000 7f17ee7fc640 1 --2- 172.21.15.16:0/793754741 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17f00d2090 0x7f17f015e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.657+0000 7f17ef7fe640 1 -- 172.21.15.16:0/793754741 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17f00d2090 msgr2=0x7f17f015e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.657+0000 7f17ef7fe640 1 --2- 172.21.15.16:0/793754741 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17f00d2090 0x7f17f015e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.657+0000 7f17ef7fe640 1 -- 172.21.15.16:0/793754741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17f00b23a0 msgr2=0x7f17f015d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.657+0000 7f17ef7fe640 1 --2- 172.21.15.16:0/793754741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17f00b23a0 0x7f17f015d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.657+0000 7f17ef7fe640 1 -- 172.21.15.16:0/793754741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f0161330 con 0x7f17f01478d0 2024-04-24T23:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.657+0000 7f17ef7fe640 1 --2- 172.21.15.16:0/793754741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17f01478d0 0x7f17f015e8a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f17e80098c0 tx=0x7f17e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.657+0000 7f17e77fe640 1 -- 172.21.15.16:0/793754741 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8041070 con 0x7f17f01478d0 2024-04-24T23:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.657+0000 7f17e77fe640 1 -- 172.21.15.16:0/793754741 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17e803c870 con 0x7f17f01478d0 2024-04-24T23:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.657+0000 7f17f53dc640 1 -- 172.21.15.16:0/793754741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f015dca0 con 0x7f17f01478d0 2024-04-24T23:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.657+0000 7f17e77fe640 1 -- 172.21.15.16:0/793754741 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8037040 con 0x7f17f01478d0 2024-04-24T23:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.657+0000 7f17f53dc640 1 -- 172.21.15.16:0/793754741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f015f1b0 con 0x7f17f01478d0 2024-04-24T23:48:59.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.658+0000 7f17f53dc640 1 -- 172.21.15.16:0/793754741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17f0068f90 con 0x7f17f01478d0 2024-04-24T23:48:59.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.658+0000 7f17e77fe640 1 -- 172.21.15.16:0/793754741 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f17e8044070 con 0x7f17f01478d0 2024-04-24T23:48:59.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.659+0000 7f17e77fe640 1 --2- 172.21.15.16:0/793754741 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f17d006be30 0x7f17d006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:48:59.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.659+0000 7f17e77fe640 1 -- 172.21.15.16:0/793754741 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6500+0+0 (secure 0 0 0) 0x7f17e80abd00 con 0x7f17f01478d0 2024-04-24T23:48:59.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.659+0000 7f17eeffd640 1 --2- 172.21.15.16:0/793754741 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f17d006be30 0x7f17d006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:48:59.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.660+0000 7f17eeffd640 1 --2- 172.21.15.16:0/793754741 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f17d006be30 0x7f17d006e2f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009560 tx=0x7f17d8009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:48:59.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.663+0000 7f17e77fe640 1 -- 172.21.15.16:0/793754741 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f17e8079e60 con 0x7f17f01478d0 2024-04-24T23:48:59.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:48:59.886+0000 7f17f53dc640 1 -- 172.21.15.16:0/793754741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f17f00d0c10 con 0x7f17f01478d0 2024-04-24T23:49:00.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.505+0000 7f17e77fe640 1 -- 172.21.15.16:0/793754741 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v6) ==== 137+0+0 (secure 0 0 0) 0x7f17e807e160 con 0x7f17f01478d0 2024-04-24T23:49:00.505 INFO:teuthology.orchestra.run.smithi016.stderr:added data pool 4 to fsmap 2024-04-24T23:49:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.509+0000 7f17f53dc640 1 -- 172.21.15.16:0/793754741 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f17d006be30 msgr2=0x7f17d006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.509+0000 7f17f53dc640 1 --2- 172.21.15.16:0/793754741 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f17d006be30 0x7f17d006e2f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f17d8009560 tx=0x7f17d8009190 comp rx=0 tx=0).stop 2024-04-24T23:49:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.509+0000 7f17f53dc640 1 -- 172.21.15.16:0/793754741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17f01478d0 msgr2=0x7f17f015e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.509+0000 7f17f53dc640 1 --2- 172.21.15.16:0/793754741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17f01478d0 0x7f17f015e8a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f17e80098c0 tx=0x7f17e80046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.509+0000 7f17f53dc640 1 -- 172.21.15.16:0/793754741 shutdown_connections 2024-04-24T23:49:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.509+0000 7f17f53dc640 1 --2- 172.21.15.16:0/793754741 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f17d006be30 0x7f17d006e2f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.509+0000 7f17f53dc640 1 --2- 172.21.15.16:0/793754741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17f01478d0 0x7f17f015e8a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.509+0000 7f17f53dc640 1 --2- 172.21.15.16:0/793754741 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17f00d2090 0x7f17f015e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.509+0000 7f17f53dc640 1 --2- 172.21.15.16:0/793754741 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17f00b23a0 0x7f17f015d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.509+0000 7f17f53dc640 1 -- 172.21.15.16:0/793754741 >> 172.21.15.16:0/793754741 conn(0x7f17f006b2c0 msgr2=0x7f17f0070090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.509+0000 7f17f53dc640 1 -- 172.21.15.16:0/793754741 shutdown_connections 2024-04-24T23:49:00.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.509+0000 7f17f53dc640 1 -- 172.21.15.16:0/793754741 wait complete. 2024-04-24T23:49:00.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:49:00.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.630+0000 7f567cdf9640 1 Processor -- start 2024-04-24T23:49:00.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.630+0000 7f567cdf9640 1 -- start start 2024-04-24T23:49:00.637 INFO:tasks.ceph.mds.b.smithi156.stderr:2024-04-24T23:49:00.633+0000 7f4c04643640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T23:49:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f567cdf9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5678072110 0x7f5678161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f567cdf9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56780d0530 0x7f5678161fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f567cdf9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56780b6a70 con 0x7f567815df20 2024-04-24T23:49:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f567cdf9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f56780cf460 con 0x7f5678072110 2024-04-24T23:49:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f567cdf9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f56780b6820 con 0x7f56780d0530 2024-04-24T23:49:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f567659c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f567815df20 0x7f5678071a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44458/0 (socket says 172.21.15.16:44458) 2024-04-24T23:49:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f567659c640 1 -- 172.21.15.16:0/4248716994 learned_addr learned my addr 172.21.15.16:0/4248716994 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f566ffff640 1 --2- 172.21.15.16:0/4248716994 >> v2:172.21.15.156:3300/0 conn(0x7f5678072110 0x7f5678161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5676d9d640 1 --2- 172.21.15.16:0/4248716994 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56780d0530 0x7f5678161fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5675d9b640 1 -- 172.21.15.16:0/4248716994 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3229832896 0 0) 0x7f56780b6a70 con 0x7f567815df20 2024-04-24T23:49:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5675d9b640 1 -- 172.21.15.16:0/4248716994 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5664003520 con 0x7f567815df20 2024-04-24T23:49:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5675d9b640 1 -- 172.21.15.16:0/4248716994 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 747809060 0 0) 0x7f5664003520 con 0x7f567815df20 2024-04-24T23:49:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5675d9b640 1 -- 172.21.15.16:0/4248716994 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56780d0530 msgr2=0x7f5678161fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5675d9b640 1 --2- 172.21.15.16:0/4248716994 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56780d0530 0x7f5678161fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5675d9b640 1 -- 172.21.15.16:0/4248716994 >> v2:172.21.15.156:3300/0 conn(0x7f5678072110 msgr2=0x7f5678161a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5675d9b640 1 --2- 172.21.15.16:0/4248716994 >> v2:172.21.15.156:3300/0 conn(0x7f5678072110 0x7f5678161a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5675d9b640 1 -- 172.21.15.16:0/4248716994 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56780ce6a0 con 0x7f567815df20 2024-04-24T23:49:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5675d9b640 1 -- 172.21.15.16:0/4248716994 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f5668001670 con 0x7f567815df20 2024-04-24T23:49:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f566ffff640 1 --2- 172.21.15.16:0/4248716994 >> v2:172.21.15.156:3300/0 conn(0x7f5678072110 0x7f5678161a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5675d9b640 1 -- 172.21.15.16:0/4248716994 >> v1:172.21.15.16:6789/0 conn(0x7f567815df20 legacy=0x7f5678071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5675d9b640 1 --2- 172.21.15.16:0/4248716994 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56780d0530 0x7f56640039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5675d9b640 1 --2- 172.21.15.16:0/4248716994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5664003f30 0x7f5664004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5675d9b640 1 --2- 172.21.15.16:0/4248716994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5664004d40 0x7f56640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.631+0000 7f5675d9b640 1 -- 172.21.15.16:0/4248716994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f56780cf460 con 0x7f5664003f30 2024-04-24T23:49:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.632+0000 7f5675d9b640 1 -- 172.21.15.16:0/4248716994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f56780b6820 con 0x7f5664004d40 2024-04-24T23:49:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.632+0000 7f5675d9b640 1 -- 172.21.15.16:0/4248716994 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f566400b9c0 con 0x7f56780d0530 2024-04-24T23:49:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.632+0000 7f566ffff640 1 --2- 172.21.15.16:0/4248716994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5664003f30 0x7f5664004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.632+0000 7f567659c640 1 --2- 172.21.15.16:0/4248716994 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56780d0530 0x7f56640039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.632+0000 7f5676d9d640 1 --2- 172.21.15.16:0/4248716994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5664004d40 0x7f56640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.632+0000 7f566ffff640 1 -- 172.21.15.16:0/4248716994 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56780d0530 msgr2=0x7f56640039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.632+0000 7f566ffff640 1 --2- 172.21.15.16:0/4248716994 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56780d0530 0x7f56640039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.632+0000 7f566ffff640 1 -- 172.21.15.16:0/4248716994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5664004d40 msgr2=0x7f56640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.632+0000 7f566ffff640 1 --2- 172.21.15.16:0/4248716994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5664004d40 0x7f56640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.632+0000 7f566ffff640 1 -- 172.21.15.16:0/4248716994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5664003520 con 0x7f5664003f30 2024-04-24T23:49:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.632+0000 7f567659c640 1 --2- 172.21.15.16:0/4248716994 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f56780d0530 0x7f56640039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.633+0000 7f566ffff640 1 --2- 172.21.15.16:0/4248716994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5664003f30 0x7f5664004800 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5678073820 tx=0x7f5664005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4232607b69f01f0d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.633+0000 7f5675d9b640 1 -- 172.21.15.16:0/4248716994 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5660009860 con 0x7f5664003f30 2024-04-24T23:49:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.633+0000 7f5675d9b640 1 -- 172.21.15.16:0/4248716994 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5660004020 con 0x7f5664003f30 2024-04-24T23:49:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.633+0000 7f5675d9b640 1 -- 172.21.15.16:0/4248716994 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56600043f0 con 0x7f5664003f30 2024-04-24T23:49:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.633+0000 7f567cdf9640 1 -- 172.21.15.16:0/4248716994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5664003f30 msgr2=0x7f5664004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.633+0000 7f567cdf9640 1 --2- 172.21.15.16:0/4248716994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5664003f30 0x7f5664004800 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5678073820 tx=0x7f5664005830 comp rx=0 tx=0).stop 2024-04-24T23:49:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.633+0000 7f5676d9d640 1 -- 172.21.15.16:0/4248716994 reap_dead start 2024-04-24T23:49:00.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.633+0000 7f567cdf9640 1 -- 172.21.15.16:0/4248716994 shutdown_connections 2024-04-24T23:49:00.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.633+0000 7f567cdf9640 1 -- 172.21.15.16:0/4248716994 >> 172.21.15.16:0/4248716994 conn(0x7f56780d0ac0 msgr2=0x7f56780d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:00.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.633+0000 7f567cdf9640 1 -- 172.21.15.16:0/4248716994 shutdown_connections 2024-04-24T23:49:00.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.633+0000 7f567cdf9640 1 -- 172.21.15.16:0/4248716994 wait complete. 2024-04-24T23:49:00.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.635+0000 7f567cdf9640 1 Processor -- start 2024-04-24T23:49:00.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.635+0000 7f567cdf9640 1 -- start start 2024-04-24T23:49:00.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.635+0000 7f567cdf9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5664003f30 0x7f56780b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.635+0000 7f567cdf9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5678150890 0x7f56780b2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.635+0000 7f567cdf9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5678152fd0 0x7f5678153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.635+0000 7f567cdf9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f56780cef40 con 0x7f5678152fd0 2024-04-24T23:49:00.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.635+0000 7f567cdf9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f56780cec60 con 0x7f5678150890 2024-04-24T23:49:00.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.635+0000 7f567cdf9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f56780b4030 con 0x7f5664003f30 2024-04-24T23:49:00.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.635+0000 7f5676d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5678152fd0 0x7f5678153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f5676d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5678152fd0 0x7f5678153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43374/0 (socket says 172.21.15.16:43374) 2024-04-24T23:49:00.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f5676d9d640 1 -- 172.21.15.16:0/4080363007 learned_addr learned my addr 172.21.15.16:0/4080363007 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:00.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f567659c640 1 --2- 172.21.15.16:0/4080363007 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5664003f30 0x7f56780b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f566ffff640 1 --2- 172.21.15.16:0/4080363007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5678150890 0x7f56780b2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f5676d9d640 1 -- 172.21.15.16:0/4080363007 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5664003f30 msgr2=0x7f56780b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:00.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f5676d9d640 1 --2- 172.21.15.16:0/4080363007 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5664003f30 0x7f56780b27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f5676d9d640 1 -- 172.21.15.16:0/4080363007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5678150890 msgr2=0x7f56780b2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:00.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f5676d9d640 1 --2- 172.21.15.16:0/4080363007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5678150890 0x7f56780b2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f5676d9d640 1 -- 172.21.15.16:0/4080363007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5678153d00 con 0x7f5678152fd0 2024-04-24T23:49:00.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f5676d9d640 1 --2- 172.21.15.16:0/4080363007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5678152fd0 0x7f5678153450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f56780731a0 tx=0x7f56700115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:00.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f566effd640 1 -- 172.21.15.16:0/4080363007 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5670011de0 con 0x7f5678152fd0 2024-04-24T23:49:00.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f566effd640 1 -- 172.21.15.16:0/4080363007 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f567001b070 con 0x7f5678152fd0 2024-04-24T23:49:00.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f567cdf9640 1 -- 172.21.15.16:0/4080363007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5678151150 con 0x7f5678152fd0 2024-04-24T23:49:00.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f566effd640 1 -- 172.21.15.16:0/4080363007 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5670003990 con 0x7f5678152fd0 2024-04-24T23:49:00.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.636+0000 7f567cdf9640 1 -- 172.21.15.16:0/4080363007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5678151710 con 0x7f5678152fd0 2024-04-24T23:49:00.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.637+0000 7f567cdf9640 1 -- 172.21.15.16:0/4080363007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f567806f8f0 con 0x7f5678152fd0 2024-04-24T23:49:00.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.638+0000 7f566effd640 1 -- 172.21.15.16:0/4080363007 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5670005d10 con 0x7f5678152fd0 2024-04-24T23:49:00.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.638+0000 7f566effd640 1 --2- 172.21.15.16:0/4080363007 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f565806be60 0x7f565806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.638+0000 7f566effd640 1 -- 172.21.15.16:0/4080363007 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f567008b2e0 con 0x7f5678152fd0 2024-04-24T23:49:00.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.639+0000 7f567659c640 1 --2- 172.21.15.16:0/4080363007 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f565806be60 0x7f565806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.639+0000 7f567659c640 1 --2- 172.21.15.16:0/4080363007 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f565806be60 0x7f565806e320 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5668001eb0 tx=0x7f5668007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:00.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.643+0000 7f566effd640 1 -- 172.21.15.16:0/4080363007 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5670059390 con 0x7f5678152fd0 2024-04-24T23:49:00.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.864+0000 7f567cdf9640 1 -- 172.21.15.16:0/4080363007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f567806d0e0 con 0x7f5678152fd0 2024-04-24T23:49:00.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.864+0000 7f566effd640 1 -- 172.21.15.16:0/4080363007 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5180 (secure 0 0 0) 0x7f567005d690 con 0x7f5678152fd0 2024-04-24T23:49:00.864 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:00.865 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1623032341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1623032341},{"type":"v1","addr":"172.21.15.156:6836","nonce":1623032341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3530738236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3530738236},{"type":"v1","addr":"172.21.15.16:6835","nonce":3530738236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2153063131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2153063131},{"type":"v1","addr":"172.21.15.16:6834","nonce":2153063131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:48:54.852787+0000","modified":"2024-04-24T23:48:59.888183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/289127714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":289127714},{"type":"v1","addr":"172.21.15.156:6837","nonce":289127714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-24T23:49:00.865 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 6 2024-04-24T23:49:00.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.866+0000 7f567cdf9640 1 -- 172.21.15.16:0/4080363007 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f565806be60 msgr2=0x7f565806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:00.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.866+0000 7f567cdf9640 1 --2- 172.21.15.16:0/4080363007 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f565806be60 0x7f565806e320 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5668001eb0 tx=0x7f5668007040 comp rx=0 tx=0).stop 2024-04-24T23:49:00.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.866+0000 7f567cdf9640 1 -- 172.21.15.16:0/4080363007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5678152fd0 msgr2=0x7f5678153450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:00.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.866+0000 7f567cdf9640 1 --2- 172.21.15.16:0/4080363007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5678152fd0 0x7f5678153450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f56780731a0 tx=0x7f56700115a0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.867+0000 7f567cdf9640 1 -- 172.21.15.16:0/4080363007 shutdown_connections 2024-04-24T23:49:00.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.867+0000 7f567cdf9640 1 --2- 172.21.15.16:0/4080363007 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f565806be60 0x7f565806e320 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.867+0000 7f567cdf9640 1 --2- 172.21.15.16:0/4080363007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5678152fd0 0x7f5678153450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.867+0000 7f567cdf9640 1 --2- 172.21.15.16:0/4080363007 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5678150890 0x7f56780b2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.867+0000 7f567cdf9640 1 --2- 172.21.15.16:0/4080363007 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5664003f30 0x7f56780b27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.867+0000 7f567cdf9640 1 -- 172.21.15.16:0/4080363007 >> 172.21.15.16:0/4080363007 conn(0x7f56780d0ac0 msgr2=0x7f56780b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:00.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.867+0000 7f567cdf9640 1 -- 172.21.15.16:0/4080363007 shutdown_connections 2024-04-24T23:49:00.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.867+0000 7f567cdf9640 1 -- 172.21.15.16:0/4080363007 wait complete. 2024-04-24T23:49:00.877 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:00.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.978+0000 7f6ef629e640 1 Processor -- start 2024-04-24T23:49:00.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.978+0000 7f6ef629e640 1 -- start start 2024-04-24T23:49:00.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6ef629e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6ef00ce5f0 0x7f6ef00cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6ef629e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ef00d05a0 con 0x7f6ef015dfa0 2024-04-24T23:49:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6ef629e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ef00d0880 con 0x7f6ef00ccad0 2024-04-24T23:49:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6ef629e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f6ef00b7360 con 0x7f6ef00ce5f0 2024-04-24T23:49:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6eeffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6ef015dfa0 0x7f6ef00d0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44470/0 (socket says 172.21.15.16:44470) 2024-04-24T23:49:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6eeffff640 1 -- 172.21.15.16:0/55535878 learned_addr learned my addr 172.21.15.16:0/55535878 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6eef7fe640 1 --2- 172.21.15.16:0/55535878 >> v2:172.21.15.156:3300/0 conn(0x7f6ef00ce5f0 0x7f6ef00cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6eeeffd640 1 -- 172.21.15.16:0/55535878 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3731829771 0 0) 0x7f6ef00d05a0 con 0x7f6ef015dfa0 2024-04-24T23:49:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6eeeffd640 1 -- 172.21.15.16:0/55535878 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ed8003520 con 0x7f6ef015dfa0 2024-04-24T23:49:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6eeeffd640 1 -- 172.21.15.16:0/55535878 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 746853666 0 0) 0x7f6ed8003520 con 0x7f6ef015dfa0 2024-04-24T23:49:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6eeeffd640 1 -- 172.21.15.16:0/55535878 >> v2:172.21.15.156:3300/0 conn(0x7f6ef00ce5f0 msgr2=0x7f6ef00cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6eeeffd640 1 --2- 172.21.15.16:0/55535878 >> v2:172.21.15.156:3300/0 conn(0x7f6ef00ce5f0 0x7f6ef00cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6eeeffd640 1 -- 172.21.15.16:0/55535878 >> v1:172.21.15.156:6789/0 conn(0x7f6ef00ccad0 legacy=0x7f6ef00ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6eeeffd640 1 -- 172.21.15.16:0/55535878 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef00d0b60 con 0x7f6ef015dfa0 2024-04-24T23:49:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.979+0000 7f6eeeffd640 1 -- 172.21.15.16:0/55535878 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f6ee4001670 con 0x7f6ef015dfa0 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6eeeffd640 1 -- 172.21.15.16:0/55535878 >> v1:172.21.15.16:6789/0 conn(0x7f6ef015dfa0 legacy=0x7f6ef00d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6eeeffd640 1 --2- 172.21.15.16:0/55535878 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ed80039b0 0x7f6ed8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6eeeffd640 1 --2- 172.21.15.16:0/55535878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ed8004800 0x7f6ed8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6eeeffd640 1 --2- 172.21.15.16:0/55535878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed80051c0 0x7f6ed800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6eeeffd640 1 -- 172.21.15.16:0/55535878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed800c030 con 0x7f6ed80051c0 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6eeeffd640 1 -- 172.21.15.16:0/55535878 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6ed800c1e0 con 0x7f6ed8004800 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6eeeffd640 1 -- 172.21.15.16:0/55535878 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6ed800c3e0 con 0x7f6ed80039b0 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6ef483b640 1 --2- 172.21.15.16:0/55535878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed80051c0 0x7f6ed800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6eeffff640 1 --2- 172.21.15.16:0/55535878 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ed80039b0 0x7f6ed8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6eef7fe640 1 --2- 172.21.15.16:0/55535878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ed8004800 0x7f6ed8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6ef483b640 1 -- 172.21.15.16:0/55535878 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ed80039b0 msgr2=0x7f6ed8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6ef483b640 1 --2- 172.21.15.16:0/55535878 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ed80039b0 0x7f6ed8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6ef483b640 1 -- 172.21.15.16:0/55535878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ed8004800 msgr2=0x7f6ed8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6ef483b640 1 --2- 172.21.15.16:0/55535878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ed8004800 0x7f6ed8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.980+0000 7f6ef483b640 1 -- 172.21.15.16:0/55535878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ed8003520 con 0x7f6ed80051c0 2024-04-24T23:49:00.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.981+0000 7f6ef483b640 1 -- 172.21.15.16:0/55535878 reap_dead start 2024-04-24T23:49:00.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.981+0000 7f6ef483b640 1 --2- 172.21.15.16:0/55535878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed80051c0 0x7f6ed800b9c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8001e50 tx=0x7f6ed8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c8448a1af83100 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:00.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.981+0000 7f6eeeffd640 1 -- 172.21.15.16:0/55535878 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8001a70 con 0x7f6ed80051c0 2024-04-24T23:49:00.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.981+0000 7f6eeeffd640 1 -- 172.21.15.16:0/55535878 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee8003970 con 0x7f6ed80051c0 2024-04-24T23:49:00.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.981+0000 7f6eeeffd640 1 -- 172.21.15.16:0/55535878 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8003d60 con 0x7f6ed80051c0 2024-04-24T23:49:00.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.981+0000 7f6ef629e640 1 -- 172.21.15.16:0/55535878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed80051c0 msgr2=0x7f6ed800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:00.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.981+0000 7f6ef629e640 1 --2- 172.21.15.16:0/55535878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed80051c0 0x7f6ed800b9c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6ee8001e50 tx=0x7f6ed8005830 comp rx=0 tx=0).stop 2024-04-24T23:49:00.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.981+0000 7f6ef629e640 1 -- 172.21.15.16:0/55535878 shutdown_connections 2024-04-24T23:49:00.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.981+0000 7f6ef629e640 1 --2- 172.21.15.16:0/55535878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed80051c0 0x7f6ed800b9c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.981+0000 7f6ef629e640 1 -- 172.21.15.16:0/55535878 >> 172.21.15.16:0/55535878 conn(0x7f6ef006b4d0 msgr2=0x7f6ef00b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:00.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.981+0000 7f6ef629e640 1 -- 172.21.15.16:0/55535878 shutdown_connections 2024-04-24T23:49:00.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.981+0000 7f6ef629e640 1 -- 172.21.15.16:0/55535878 wait complete. 2024-04-24T23:49:00.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.983+0000 7f6ef629e640 1 Processor -- start 2024-04-24T23:49:00.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.983+0000 7f6ef629e640 1 -- start start 2024-04-24T23:49:00.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6ef629e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed80051c0 0x7f6ef0153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6ef629e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ef00c9d00 0x7f6ef00ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6ef629e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ef00ca6c0 0x7f6ef0159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6ef629e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef015e380 con 0x7f6ed80051c0 2024-04-24T23:49:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6ef629e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef00cc640 con 0x7f6ef00ca6c0 2024-04-24T23:49:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6ef629e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6ef00d01c0 con 0x7f6ef00c9d00 2024-04-24T23:49:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6eeffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed80051c0 0x7f6ef0153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6eeffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed80051c0 0x7f6ef0153760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43402/0 (socket says 172.21.15.16:43402) 2024-04-24T23:49:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6eeffff640 1 -- 172.21.15.16:0/979128259 learned_addr learned my addr 172.21.15.16:0/979128259 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6ef483b640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ef00ca6c0 0x7f6ef0159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6eef7fe640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ef00c9d00 0x7f6ef00ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6eeffff640 1 -- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ef00c9d00 msgr2=0x7f6ef00ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6eeffff640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ef00c9d00 0x7f6ef00ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6eeffff640 1 -- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ef00ca6c0 msgr2=0x7f6ef0159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6eeffff640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ef00ca6c0 0x7f6ef0159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6eeffff640 1 -- 172.21.15.16:0/979128259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef0158310 con 0x7f6ed80051c0 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6eef7fe640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ef00c9d00 0x7f6ef00ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6ef483b640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ef00ca6c0 0x7f6ef0159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6eeffff640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed80051c0 0x7f6ef0153760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4007340 tx=0x7f6ee4007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6eecff9640 1 -- 172.21.15.16:0/979128259 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee400f940 con 0x7f6ed80051c0 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6eecff9640 1 -- 172.21.15.16:0/979128259 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee4018070 con 0x7f6ed80051c0 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6ef629e640 1 -- 172.21.15.16:0/979128259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef00cab40 con 0x7f6ed80051c0 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6eecff9640 1 -- 172.21.15.16:0/979128259 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee400fd40 con 0x7f6ed80051c0 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.984+0000 7f6ef629e640 1 -- 172.21.15.16:0/979128259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef0158860 con 0x7f6ed80051c0 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.985+0000 7f6ef629e640 1 -- 172.21.15.16:0/979128259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ef006bea0 con 0x7f6ed80051c0 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.986+0000 7f6eecff9640 1 -- 172.21.15.16:0/979128259 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6ee401d460 con 0x7f6ed80051c0 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.986+0000 7f6eecff9640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6edc06be60 0x7f6edc06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.986+0000 7f6eecff9640 1 -- 172.21.15.16:0/979128259 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6ee408b150 con 0x7f6ed80051c0 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.986+0000 7f6eef7fe640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6edc06be60 0x7f6edc06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.987+0000 7f6eef7fe640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6edc06be60 0x7f6edc06e320 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0160cd0 tx=0x7f6ee0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:00.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:00.991+0000 7f6eecff9640 1 -- 172.21.15.16:0/979128259 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6ee4059910 con 0x7f6ed80051c0 2024-04-24T23:49:01.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.175+0000 7f6ef629e640 1 -- 172.21.15.16:0/979128259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6ef004f5e0 con 0x7f6ed80051c0 2024-04-24T23:49:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.176+0000 7f6eecff9640 1 -- 172.21.15.16:0/979128259 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18502 (secure 0 0 0) 0x7f6ee4059210 con 0x7f6ed80051c0 2024-04-24T23:49:01.177 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:01.177 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:59.888015+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.178+0000 7f6ef629e640 1 -- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6edc06be60 msgr2=0x7f6edc06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.178+0000 7f6ef629e640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6edc06be60 0x7f6edc06e320 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6ef0160cd0 tx=0x7f6ee0008040 comp rx=0 tx=0).stop 2024-04-24T23:49:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.178+0000 7f6ef629e640 1 -- 172.21.15.16:0/979128259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed80051c0 msgr2=0x7f6ef0153760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.178+0000 7f6ef629e640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed80051c0 0x7f6ef0153760 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6ee4007340 tx=0x7f6ee4007800 comp rx=0 tx=0).stop 2024-04-24T23:49:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.178+0000 7f6ef629e640 1 -- 172.21.15.16:0/979128259 shutdown_connections 2024-04-24T23:49:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.178+0000 7f6ef629e640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6edc06be60 0x7f6edc06e320 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.178+0000 7f6ef629e640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ef00ca6c0 0x7f6ef0159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.178+0000 7f6ef629e640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ef00c9d00 0x7f6ef00ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.178+0000 7f6ef629e640 1 --2- 172.21.15.16:0/979128259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ed80051c0 0x7f6ef0153760 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.178+0000 7f6ef629e640 1 -- 172.21.15.16:0/979128259 >> 172.21.15.16:0/979128259 conn(0x7f6ef006b4d0 msgr2=0x7f6ef015e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.178+0000 7f6ef629e640 1 -- 172.21.15.16:0/979128259 shutdown_connections 2024-04-24T23:49:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.178+0000 7f6ef629e640 1 -- 172.21.15.16:0/979128259 wait complete. 2024-04-24T23:49:01.190 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:01.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.291+0000 7f89c848c640 1 Processor -- start 2024-04-24T23:49:01.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.291+0000 7f89c848c640 1 -- start start 2024-04-24T23:49:01.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c848c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f89c015ed10 0x7f89c015e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c848c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89c00cff10 0x7f89c006b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c848c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89c00d0370 con 0x7f89c015e930 2024-04-24T23:49:01.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c848c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f89c006bcd0 con 0x7f89c015ed10 2024-04-24T23:49:01.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c848c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f89c006bfb0 con 0x7f89c00cff10 2024-04-24T23:49:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c5a27640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f89c015ed10 0x7f89c015e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c6228640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f89c015e930 0x7f89c015dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44472/0 (socket says 172.21.15.16:44472) 2024-04-24T23:49:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c6228640 1 -- 172.21.15.16:0/3158257780 learned_addr learned my addr 172.21.15.16:0/3158257780 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c6a29640 1 --2- 172.21.15.16:0/3158257780 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89c00cff10 0x7f89c006b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c5a27640 1 -- 172.21.15.16:0/3158257780 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89c00cff10 msgr2=0x7f89c006b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c5a27640 1 --2- 172.21.15.16:0/3158257780 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89c00cff10 0x7f89c006b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c5a27640 1 -- 172.21.15.16:0/3158257780 >> v1:172.21.15.16:6789/0 conn(0x7f89c015e930 legacy=0x7f89c015dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c5a27640 1 -- 172.21.15.16:0/3158257780 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c006c290 con 0x7f89c015ed10 2024-04-24T23:49:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c5226640 1 -- 172.21.15.16:0/3158257780 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1861159365 0 0) 0x7f89bc001110 con 0x7f89c015e930 2024-04-24T23:49:01.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.292+0000 7f89c6a29640 1 --2- 172.21.15.16:0/3158257780 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89c00cff10 0x7f89c006b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:01.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.293+0000 7f89c5a27640 1 --2- 172.21.15.16:0/3158257780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89c015ed10 0x7f89c015e1d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f89b00097a0 tx=0x7f89b002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b5f8ba04e11dba0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:01.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.293+0000 7f89c5226640 1 -- 172.21.15.16:0/3158257780 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0037070 con 0x7f89c015ed10 2024-04-24T23:49:01.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.293+0000 7f89c5226640 1 -- 172.21.15.16:0/3158257780 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b0032440 con 0x7f89c015ed10 2024-04-24T23:49:01.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.293+0000 7f89c5226640 1 -- 172.21.15.16:0/3158257780 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b0032810 con 0x7f89c015ed10 2024-04-24T23:49:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.293+0000 7f89c848c640 1 -- 172.21.15.16:0/3158257780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89c015ed10 msgr2=0x7f89c015e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.293+0000 7f89c848c640 1 --2- 172.21.15.16:0/3158257780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89c015ed10 0x7f89c015e1d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f89b00097a0 tx=0x7f89b002a180 comp rx=0 tx=0).stop 2024-04-24T23:49:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.294+0000 7f89c848c640 1 -- 172.21.15.16:0/3158257780 shutdown_connections 2024-04-24T23:49:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.294+0000 7f89c848c640 1 --2- 172.21.15.16:0/3158257780 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89c00cff10 0x7f89c006b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.294+0000 7f89c848c640 1 --2- 172.21.15.16:0/3158257780 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89c015ed10 0x7f89c015e1d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.294+0000 7f89c848c640 1 -- 172.21.15.16:0/3158257780 >> 172.21.15.16:0/3158257780 conn(0x7f89c00d4050 msgr2=0x7f89c00d4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.294+0000 7f89c848c640 1 -- 172.21.15.16:0/3158257780 shutdown_connections 2024-04-24T23:49:01.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.294+0000 7f89c848c640 1 -- 172.21.15.16:0/3158257780 wait complete. 2024-04-24T23:49:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.295+0000 7f89c848c640 1 Processor -- start 2024-04-24T23:49:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.295+0000 7f89c848c640 1 -- start start 2024-04-24T23:49:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.295+0000 7f89c848c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89c00cff10 0x7f89c0147410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.295+0000 7f89c848c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89c015e930 0x7f89c015f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.295+0000 7f89c848c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89c015ed10 0x7f89c015f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.295+0000 7f89c848c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f89c00d1bf0 con 0x7f89c015ed10 2024-04-24T23:49:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.295+0000 7f89c848c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f89c006c850 con 0x7f89c00cff10 2024-04-24T23:49:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.295+0000 7f89c848c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f89c00d1d70 con 0x7f89c015e930 2024-04-24T23:49:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89c6a29640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89c015ed10 0x7f89c015f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89c6a29640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89c015ed10 0x7f89c015f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43432/0 (socket says 172.21.15.16:43432) 2024-04-24T23:49:01.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89c5a27640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89c015e930 0x7f89c015f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89c6a29640 1 -- 172.21.15.16:0/306302276 learned_addr learned my addr 172.21.15.16:0/306302276 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89c6228640 1 --2- 172.21.15.16:0/306302276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89c00cff10 0x7f89c0147410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89c6a29640 1 -- 172.21.15.16:0/306302276 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89c015e930 msgr2=0x7f89c015f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89c6a29640 1 --2- 172.21.15.16:0/306302276 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89c015e930 0x7f89c015f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89c6a29640 1 -- 172.21.15.16:0/306302276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89c00cff10 msgr2=0x7f89c0147410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89c6a29640 1 --2- 172.21.15.16:0/306302276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89c00cff10 0x7f89c0147410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89c6a29640 1 -- 172.21.15.16:0/306302276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89c015b1d0 con 0x7f89c015ed10 2024-04-24T23:49:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89c6a29640 1 --2- 172.21.15.16:0/306302276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89c015ed10 0x7f89c015f680 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f89b800cef0 tx=0x7f89b800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89b6ffd640 1 -- 172.21.15.16:0/306302276 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b801a070 con 0x7f89c015ed10 2024-04-24T23:49:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89b6ffd640 1 -- 172.21.15.16:0/306302276 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89b800d860 con 0x7f89c015ed10 2024-04-24T23:49:01.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89c848c640 1 -- 172.21.15.16:0/306302276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89c01629c0 con 0x7f89c015ed10 2024-04-24T23:49:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89b6ffd640 1 -- 172.21.15.16:0/306302276 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89b800dc50 con 0x7f89c015ed10 2024-04-24T23:49:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.296+0000 7f89c848c640 1 -- 172.21.15.16:0/306302276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89c00c8d20 con 0x7f89c015ed10 2024-04-24T23:49:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.297+0000 7f89c848c640 1 -- 172.21.15.16:0/306302276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89c006dfe0 con 0x7f89c015ed10 2024-04-24T23:49:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.298+0000 7f89b6ffd640 1 -- 172.21.15.16:0/306302276 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f89b801d070 con 0x7f89c015ed10 2024-04-24T23:49:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.298+0000 7f89b6ffd640 1 --2- 172.21.15.16:0/306302276 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f89a406be30 0x7f89a406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.298+0000 7f89b6ffd640 1 -- 172.21.15.16:0/306302276 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f89b808ba80 con 0x7f89c015ed10 2024-04-24T23:49:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.299+0000 7f89c6228640 1 --2- 172.21.15.16:0/306302276 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f89a406be30 0x7f89a406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.299+0000 7f89c6228640 1 --2- 172.21.15.16:0/306302276 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f89a406be30 0x7f89a406e2f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f89bc000f60 tx=0x7f89bc007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:01.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.302+0000 7f89b6ffd640 1 -- 172.21.15.16:0/306302276 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f89b808de20 con 0x7f89c015ed10 2024-04-24T23:49:01.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.485+0000 7f89c848c640 1 -- 172.21.15.16:0/306302276 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f89c004f5e0 con 0x7f89c015ed10 2024-04-24T23:49:01.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.487+0000 7f89b6ffd640 1 -- 172.21.15.16:0/306302276 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18502 (secure 0 0 0) 0x7f89b8059ac0 con 0x7f89c015ed10 2024-04-24T23:49:01.487 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:01.487 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:59.888015+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:01.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.488+0000 7f89c848c640 1 -- 172.21.15.16:0/306302276 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f89a406be30 msgr2=0x7f89a406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.488+0000 7f89c848c640 1 --2- 172.21.15.16:0/306302276 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f89a406be30 0x7f89a406e2f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f89bc000f60 tx=0x7f89bc007b50 comp rx=0 tx=0).stop 2024-04-24T23:49:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.488+0000 7f89c848c640 1 -- 172.21.15.16:0/306302276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89c015ed10 msgr2=0x7f89c015f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.488+0000 7f89c848c640 1 --2- 172.21.15.16:0/306302276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89c015ed10 0x7f89c015f680 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f89b800cef0 tx=0x7f89b800a710 comp rx=0 tx=0).stop 2024-04-24T23:49:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.488+0000 7f89c848c640 1 -- 172.21.15.16:0/306302276 shutdown_connections 2024-04-24T23:49:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.488+0000 7f89c848c640 1 --2- 172.21.15.16:0/306302276 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f89a406be30 0x7f89a406e2f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.488+0000 7f89c848c640 1 --2- 172.21.15.16:0/306302276 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89c015ed10 0x7f89c015f680 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.488+0000 7f89c848c640 1 --2- 172.21.15.16:0/306302276 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f89c015e930 0x7f89c015f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.488+0000 7f89c848c640 1 --2- 172.21.15.16:0/306302276 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f89c00cff10 0x7f89c0147410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.489+0000 7f89c848c640 1 -- 172.21.15.16:0/306302276 >> 172.21.15.16:0/306302276 conn(0x7f89c00d4050 msgr2=0x7f89c00cb1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.489+0000 7f89c848c640 1 -- 172.21.15.16:0/306302276 shutdown_connections 2024-04-24T23:49:01.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.489+0000 7f89c848c640 1 -- 172.21.15.16:0/306302276 wait complete. 2024-04-24T23:49:01.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:49:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.617+0000 7effdec37640 1 Processor -- start 2024-04-24T23:49:01.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.617+0000 7effdec37640 1 -- start start 2024-04-24T23:49:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.618+0000 7effdec37640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7effd815e1d0 0x7effd80cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.618+0000 7effdec37640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effd806bcd0 con 0x7effd815e930 2024-04-24T23:49:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.618+0000 7effdec37640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effd806bfb0 con 0x7effd815ed10 2024-04-24T23:49:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.618+0000 7effdec37640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7effd806c290 con 0x7effd815e1d0 2024-04-24T23:49:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.618+0000 7effdd1d4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7effd815e1d0 0x7effd80cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.618+0000 7effdc9d3640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7effd815e930 0x7effd815dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44476/0 (socket says 172.21.15.16:44476) 2024-04-24T23:49:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.618+0000 7effdc9d3640 1 -- 172.21.15.16:0/2404076437 learned_addr learned my addr 172.21.15.16:0/2404076437 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:01.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.618+0000 7effdd1d4640 1 -- 172.21.15.16:0/2404076437 >> v1:172.21.15.156:6789/0 conn(0x7effd815ed10 legacy=0x7effd80d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.618+0000 7effdd1d4640 1 -- 172.21.15.16:0/2404076437 >> v1:172.21.15.16:6789/0 conn(0x7effd815e930 legacy=0x7effd815dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.618+0000 7effd77fe640 1 -- 172.21.15.16:0/2404076437 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3325618820 0 0) 0x7effd806bcd0 con 0x7effd815e930 2024-04-24T23:49:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.618+0000 7effdd1d4640 1 -- 172.21.15.16:0/2404076437 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd806c570 con 0x7effd815e1d0 2024-04-24T23:49:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.619+0000 7effdd1d4640 1 --2- 172.21.15.16:0/2404076437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effd815e1d0 0x7effd80cff10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7effd0009d90 tx=0x7effd002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98d91a64474576e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.619+0000 7effd77fe640 1 -- 172.21.15.16:0/2404076437 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0007740 con 0x7effd815e1d0 2024-04-24T23:49:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.619+0000 7effd77fe640 1 -- 172.21.15.16:0/2404076437 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effd00078e0 con 0x7effd815e1d0 2024-04-24T23:49:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.619+0000 7effd77fe640 1 -- 172.21.15.16:0/2404076437 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effd0007cb0 con 0x7effd815e1d0 2024-04-24T23:49:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.620+0000 7effdec37640 1 -- 172.21.15.16:0/2404076437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effd815e1d0 msgr2=0x7effd80cff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.620+0000 7effdec37640 1 --2- 172.21.15.16:0/2404076437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effd815e1d0 0x7effd80cff10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7effd0009d90 tx=0x7effd002a6a0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.620+0000 7effdec37640 1 -- 172.21.15.16:0/2404076437 shutdown_connections 2024-04-24T23:49:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.620+0000 7effdec37640 1 --2- 172.21.15.16:0/2404076437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effd815e1d0 0x7effd80cff10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.620+0000 7effdec37640 1 -- 172.21.15.16:0/2404076437 >> 172.21.15.16:0/2404076437 conn(0x7effd80d4050 msgr2=0x7effd80d4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:01.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.620+0000 7effdec37640 1 -- 172.21.15.16:0/2404076437 shutdown_connections 2024-04-24T23:49:01.622 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.620+0000 7effdec37640 1 -- 172.21.15.16:0/2404076437 wait complete. 2024-04-24T23:49:01.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.621+0000 7effdec37640 1 Processor -- start 2024-04-24T23:49:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.621+0000 7effdec37640 1 -- start start 2024-04-24T23:49:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effdec37640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7effd815e1d0 0x7effd80c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effdec37640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effd815e930 0x7effd80ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effdec37640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7effd815ed10 0x7effd814c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effdec37640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7effd806b720 con 0x7effd815e930 2024-04-24T23:49:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effdec37640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7effd806b5a0 con 0x7effd815ed10 2024-04-24T23:49:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effdec37640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7effd806b8a0 con 0x7effd815e1d0 2024-04-24T23:49:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effd815e930 0x7effd80ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effd7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effd815e930 0x7effd80ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43442/0 (socket says 172.21.15.16:43442) 2024-04-24T23:49:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effd7fff640 1 -- 172.21.15.16:0/2683628401 learned_addr learned my addr 172.21.15.16:0/2683628401 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:01.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effdc9d3640 1 --2- 172.21.15.16:0/2683628401 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7effd815e1d0 0x7effd80c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effd7fff640 1 -- 172.21.15.16:0/2683628401 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7effd815e1d0 msgr2=0x7effd80c7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effdd1d4640 1 --2- 172.21.15.16:0/2683628401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7effd815ed10 0x7effd814c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effd7fff640 1 --2- 172.21.15.16:0/2683628401 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7effd815e1d0 0x7effd80c7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effd7fff640 1 -- 172.21.15.16:0/2683628401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7effd815ed10 msgr2=0x7effd814c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effd7fff640 1 --2- 172.21.15.16:0/2683628401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7effd815ed10 0x7effd814c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.622+0000 7effd7fff640 1 -- 172.21.15.16:0/2683628401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effd80c7f60 con 0x7effd815e930 2024-04-24T23:49:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.623+0000 7effd7fff640 1 --2- 172.21.15.16:0/2683628401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effd815e930 0x7effd80ca330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7effcc00b420 tx=0x7effcc00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.623+0000 7effd57fa640 1 -- 172.21.15.16:0/2683628401 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effcc011830 con 0x7effd815e930 2024-04-24T23:49:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.623+0000 7effd57fa640 1 -- 172.21.15.16:0/2683628401 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effcc016070 con 0x7effd815e930 2024-04-24T23:49:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.623+0000 7effdec37640 1 -- 172.21.15.16:0/2683628401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effd80c8240 con 0x7effd815e930 2024-04-24T23:49:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.623+0000 7effdec37640 1 -- 172.21.15.16:0/2683628401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effd80c8800 con 0x7effd815e930 2024-04-24T23:49:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.623+0000 7effd57fa640 1 -- 172.21.15.16:0/2683628401 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effcc011c10 con 0x7effd815e930 2024-04-24T23:49:01.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.624+0000 7effdec37640 1 -- 172.21.15.16:0/2683628401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb8005290 con 0x7effd815e930 2024-04-24T23:49:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.624+0000 7effd57fa640 1 -- 172.21.15.16:0/2683628401 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7effcc01b460 con 0x7effd815e930 2024-04-24T23:49:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.625+0000 7effd57fa640 1 --2- 172.21.15.16:0/2683628401 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7effc406be30 0x7effc406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.625+0000 7effd57fa640 1 -- 172.21.15.16:0/2683628401 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7effcc08ace0 con 0x7effd815e930 2024-04-24T23:49:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.625+0000 7effdc9d3640 1 --2- 172.21.15.16:0/2683628401 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7effc406be30 0x7effc406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.625+0000 7effdc9d3640 1 --2- 172.21.15.16:0/2683628401 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7effc406be30 0x7effc406e2f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7effc8000f60 tx=0x7effc8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:01.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.629+0000 7effd57fa640 1 -- 172.21.15.16:0/2683628401 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7effcc020020 con 0x7effd815e930 2024-04-24T23:49:01.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.849+0000 7effdec37640 1 -- 172.21.15.16:0/2683628401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7effb8005850 con 0x7effd815e930 2024-04-24T23:49:01.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.849+0000 7effd57fa640 1 -- 172.21.15.16:0/2683628401 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5180 (secure 0 0 0) 0x7effcc059350 con 0x7effd815e930 2024-04-24T23:49:01.849 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:01.849 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1623032341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1623032341},{"type":"v1","addr":"172.21.15.156:6836","nonce":1623032341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3530738236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3530738236},{"type":"v1","addr":"172.21.15.16:6835","nonce":3530738236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2153063131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2153063131},{"type":"v1","addr":"172.21.15.16:6834","nonce":2153063131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:48:54.852787+0000","modified":"2024-04-24T23:48:59.888183+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/289127714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":289127714},{"type":"v1","addr":"172.21.15.156:6837","nonce":289127714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-24T23:49:01.850 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 6 2024-04-24T23:49:01.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.851+0000 7effdec37640 1 -- 172.21.15.16:0/2683628401 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7effc406be30 msgr2=0x7effc406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.851+0000 7effdec37640 1 --2- 172.21.15.16:0/2683628401 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7effc406be30 0x7effc406e2f0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7effc8000f60 tx=0x7effc8009040 comp rx=0 tx=0).stop 2024-04-24T23:49:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.851+0000 7effdec37640 1 -- 172.21.15.16:0/2683628401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effd815e930 msgr2=0x7effd80ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.851+0000 7effdec37640 1 --2- 172.21.15.16:0/2683628401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effd815e930 0x7effd80ca330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7effcc00b420 tx=0x7effcc00b8e0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.851+0000 7effdec37640 1 -- 172.21.15.16:0/2683628401 shutdown_connections 2024-04-24T23:49:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.851+0000 7effdec37640 1 --2- 172.21.15.16:0/2683628401 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7effc406be30 0x7effc406e2f0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.851+0000 7effdec37640 1 --2- 172.21.15.16:0/2683628401 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7effd815ed10 0x7effd814c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.851+0000 7effdec37640 1 --2- 172.21.15.16:0/2683628401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7effd815e930 0x7effd80ca330 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.851+0000 7effdec37640 1 --2- 172.21.15.16:0/2683628401 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7effd815e1d0 0x7effd80c7980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.851+0000 7effdec37640 1 -- 172.21.15.16:0/2683628401 >> 172.21.15.16:0/2683628401 conn(0x7effd80d4050 msgr2=0x7effd80b6140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.851+0000 7effdec37640 1 -- 172.21.15.16:0/2683628401 shutdown_connections 2024-04-24T23:49:01.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.852+0000 7effdec37640 1 -- 172.21.15.16:0/2683628401 wait complete. 2024-04-24T23:49:01.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:01.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.963+0000 7ff5ed54c640 1 Processor -- start 2024-04-24T23:49:01.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.963+0000 7ff5ed54c640 1 -- start start 2024-04-24T23:49:01.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5ed54c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff5e815bcd0 0x7ff5e81601f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5ed54c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5e80705b0 0x7ff5e806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5ed54c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff5e80d0fb0 0x7ff5e80d13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5ed54c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff5e806f660 con 0x7ff5e815bcd0 2024-04-24T23:49:01.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5ed54c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ff5e80d1920 con 0x7ff5e80d0fb0 2024-04-24T23:49:01.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5ed54c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e81607d0 con 0x7ff5e80705b0 2024-04-24T23:49:01.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5e6ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff5e815bcd0 0x7ff5e81601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5e6ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff5e815bcd0 0x7ff5e81601f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43458/0 (socket says 172.21.15.16:43458) 2024-04-24T23:49:01.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5e6ffd640 1 -- 172.21.15.16:0/1020973113 learned_addr learned my addr 172.21.15.16:0/1020973113 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:01.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5e6ffd640 1 -- 172.21.15.16:0/1020973113 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5e80705b0 msgr2=0x7ff5e806faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:01.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5e6ffd640 1 --2- 172.21.15.16:0/1020973113 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5e80705b0 0x7ff5e806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5e6ffd640 1 -- 172.21.15.16:0/1020973113 >> v2:172.21.15.156:3300/0 conn(0x7ff5e80d0fb0 msgr2=0x7ff5e80d13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:01.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5e77fe640 1 --2- 172.21.15.16:0/1020973113 >> v2:172.21.15.156:3300/0 conn(0x7ff5e80d0fb0 0x7ff5e80d13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5e6ffd640 1 --2- 172.21.15.16:0/1020973113 >> v2:172.21.15.156:3300/0 conn(0x7ff5e80d0fb0 0x7ff5e80d13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.964+0000 7ff5e6ffd640 1 -- 172.21.15.16:0/1020973113 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e806ed30 con 0x7ff5e815bcd0 2024-04-24T23:49:01.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.965+0000 7ff5e6ffd640 1 --2- 172.21.15.16:0/1020973113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5e815bcd0 0x7ff5e81601f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80d0e20 tx=0x7ff5d802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a90c28001617ef7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:01.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.965+0000 7ff5e5ffb640 1 -- 172.21.15.16:0/1020973113 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8037070 con 0x7ff5e815bcd0 2024-04-24T23:49:01.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.965+0000 7ff5e5ffb640 1 -- 172.21.15.16:0/1020973113 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5d802ae00 con 0x7ff5e815bcd0 2024-04-24T23:49:01.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.965+0000 7ff5e5ffb640 1 -- 172.21.15.16:0/1020973113 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d8032660 con 0x7ff5e815bcd0 2024-04-24T23:49:01.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.965+0000 7ff5ed54c640 1 -- 172.21.15.16:0/1020973113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5e815bcd0 msgr2=0x7ff5e81601f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.965+0000 7ff5ed54c640 1 --2- 172.21.15.16:0/1020973113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5e815bcd0 0x7ff5e81601f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80d0e20 tx=0x7ff5d802a0b0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.965+0000 7ff5ed54c640 1 -- 172.21.15.16:0/1020973113 shutdown_connections 2024-04-24T23:49:01.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.965+0000 7ff5ed54c640 1 --2- 172.21.15.16:0/1020973113 >> v2:172.21.15.156:3300/0 conn(0x7ff5e80d0fb0 0x7ff5e80d13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.965+0000 7ff5ed54c640 1 --2- 172.21.15.16:0/1020973113 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5e80705b0 0x7ff5e806faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.965+0000 7ff5ed54c640 1 --2- 172.21.15.16:0/1020973113 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5e815bcd0 0x7ff5e81601f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.965+0000 7ff5ed54c640 1 -- 172.21.15.16:0/1020973113 >> 172.21.15.16:0/1020973113 conn(0x7ff5e806b4d0 msgr2=0x7ff5e80b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:01.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.965+0000 7ff5ed54c640 1 -- 172.21.15.16:0/1020973113 shutdown_connections 2024-04-24T23:49:01.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.965+0000 7ff5ed54c640 1 -- 172.21.15.16:0/1020973113 wait complete. 2024-04-24T23:49:01.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.967+0000 7ff5ed54c640 1 Processor -- start 2024-04-24T23:49:01.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.967+0000 7ff5ed54c640 1 -- start start 2024-04-24T23:49:01.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.967+0000 7ff5ed54c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5e80705b0 0x7ff5e8150280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.967+0000 7ff5ed54c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5e80d0fb0 0x7ff5e80ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.967+0000 7ff5ed54c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5e815bcd0 0x7ff5e80c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.967+0000 7ff5ed54c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e815c0b0 con 0x7ff5e815bcd0 2024-04-24T23:49:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.967+0000 7ff5ed54c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e806f2f0 con 0x7ff5e80d0fb0 2024-04-24T23:49:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.967+0000 7ff5ed54c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e80d09d0 con 0x7ff5e80705b0 2024-04-24T23:49:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5e77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5e815bcd0 0x7ff5e80c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5e77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5e815bcd0 0x7ff5e80c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43462/0 (socket says 172.21.15.16:43462) 2024-04-24T23:49:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5e77fe640 1 -- 172.21.15.16:0/955153164 learned_addr learned my addr 172.21.15.16:0/955153164 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5e6ffd640 1 --2- 172.21.15.16:0/955153164 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5e80705b0 0x7ff5e8150280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5e67fc640 1 --2- 172.21.15.16:0/955153164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5e80d0fb0 0x7ff5e80ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5e77fe640 1 -- 172.21.15.16:0/955153164 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5e80705b0 msgr2=0x7ff5e8150280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5e77fe640 1 --2- 172.21.15.16:0/955153164 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5e80705b0 0x7ff5e8150280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5e77fe640 1 -- 172.21.15.16:0/955153164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5e80d0fb0 msgr2=0x7ff5e80ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5e77fe640 1 --2- 172.21.15.16:0/955153164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5e80d0fb0 0x7ff5e80ca8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5e77fe640 1 -- 172.21.15.16:0/955153164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e81e6510 con 0x7ff5e815bcd0 2024-04-24T23:49:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5e77fe640 1 --2- 172.21.15.16:0/955153164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5e815bcd0 0x7ff5e80c74c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000cc30 tx=0x7ff5e000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5cf7fe640 1 -- 172.21.15.16:0/955153164 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e000ce40 con 0x7ff5e815bcd0 2024-04-24T23:49:01.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5cf7fe640 1 -- 172.21.15.16:0/955153164 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5e0003990 con 0x7ff5e815bcd0 2024-04-24T23:49:01.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5ed54c640 1 -- 172.21.15.16:0/955153164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5e81e67f0 con 0x7ff5e815bcd0 2024-04-24T23:49:01.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5cf7fe640 1 -- 172.21.15.16:0/955153164 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e0003d40 con 0x7ff5e815bcd0 2024-04-24T23:49:01.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.968+0000 7ff5ed54c640 1 -- 172.21.15.16:0/955153164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5e81e7090 con 0x7ff5e815bcd0 2024-04-24T23:49:01.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.969+0000 7ff5ed54c640 1 -- 172.21.15.16:0/955153164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5e8068320 con 0x7ff5e815bcd0 2024-04-24T23:49:01.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.970+0000 7ff5cf7fe640 1 -- 172.21.15.16:0/955153164 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff5e0003ee0 con 0x7ff5e815bcd0 2024-04-24T23:49:01.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.970+0000 7ff5cf7fe640 1 --2- 172.21.15.16:0/955153164 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff5d406be30 0x7ff5d406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:01.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.970+0000 7ff5cf7fe640 1 -- 172.21.15.16:0/955153164 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff5e008b5d0 con 0x7ff5e815bcd0 2024-04-24T23:49:01.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.970+0000 7ff5e6ffd640 1 --2- 172.21.15.16:0/955153164 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff5d406be30 0x7ff5d406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:01.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.971+0000 7ff5e6ffd640 1 --2- 172.21.15.16:0/955153164 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff5d406be30 0x7ff5d406e2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8034000 tx=0x7ff5d802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:01.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:01.974+0000 7ff5cf7fe640 1 -- 172.21.15.16:0/955153164 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff5e8068320 con 0x7ff5e815bcd0 2024-04-24T23:49:02.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.159+0000 7ff5ed54c640 1 -- 172.21.15.16:0/955153164 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff5e81e7290 con 0x7ff5e815bcd0 2024-04-24T23:49:02.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.160+0000 7ff5cf7fe640 1 -- 172.21.15.16:0/955153164 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18502 (secure 0 0 0) 0x7ff5e0014030 con 0x7ff5e815bcd0 2024-04-24T23:49:02.160 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:02.160 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:59.888015+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:02.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.162+0000 7ff5ed54c640 1 -- 172.21.15.16:0/955153164 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff5d406be30 msgr2=0x7ff5d406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.162+0000 7ff5ed54c640 1 --2- 172.21.15.16:0/955153164 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff5d406be30 0x7ff5d406e2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff5d8034000 tx=0x7ff5d802e000 comp rx=0 tx=0).stop 2024-04-24T23:49:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.162+0000 7ff5ed54c640 1 -- 172.21.15.16:0/955153164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5e815bcd0 msgr2=0x7ff5e80c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.162+0000 7ff5ed54c640 1 --2- 172.21.15.16:0/955153164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5e815bcd0 0x7ff5e80c74c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff5e000cc30 tx=0x7ff5e000a5c0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.162+0000 7ff5ed54c640 1 -- 172.21.15.16:0/955153164 shutdown_connections 2024-04-24T23:49:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.162+0000 7ff5ed54c640 1 --2- 172.21.15.16:0/955153164 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff5d406be30 0x7ff5d406e2f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.162+0000 7ff5ed54c640 1 --2- 172.21.15.16:0/955153164 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5e815bcd0 0x7ff5e80c74c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.162+0000 7ff5ed54c640 1 --2- 172.21.15.16:0/955153164 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5e80d0fb0 0x7ff5e80ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.162+0000 7ff5ed54c640 1 --2- 172.21.15.16:0/955153164 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5e80705b0 0x7ff5e8150280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.162+0000 7ff5ed54c640 1 -- 172.21.15.16:0/955153164 >> 172.21.15.16:0/955153164 conn(0x7ff5e806b4d0 msgr2=0x7ff5e8159f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.162+0000 7ff5ed54c640 1 -- 172.21.15.16:0/955153164 shutdown_connections 2024-04-24T23:49:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.162+0000 7ff5ed54c640 1 -- 172.21.15.16:0/955153164 wait complete. 2024-04-24T23:49:02.577 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T23:48:54.852787+0000', 'modified': '2024-04-24T23:48:59.888183+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4294}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4294': {'gid': 4294, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.156:6837/289127714', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6835', 'nonce': 289127714}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 289127714}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 4294, 'qdb_cluster': [4294]} 2024-04-24T23:49:02.577 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T23:49:02.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-24T23:49:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.682+0000 7f82b99a0640 1 Processor -- start 2024-04-24T23:49:02.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.682+0000 7f82b99a0640 1 -- start start 2024-04-24T23:49:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b99a0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f82b4143fd0 0x7f82b40ceba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b99a0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82b40ce410 con 0x7f82b40cf0e0 2024-04-24T23:49:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b99a0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82b415eba0 con 0x7f82b4140bb0 2024-04-24T23:49:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b99a0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f82b415e980 con 0x7f82b4143fd0 2024-04-24T23:49:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b919f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f82b40cf0e0 0x7f82b40d02d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44490/0 (socket says 172.21.15.16:44490) 2024-04-24T23:49:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b919f640 1 -- 172.21.15.16:0/1939951366 learned_addr learned my addr 172.21.15.16:0/1939951366 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b3fff640 1 --2- 172.21.15.16:0/1939951366 >> v2:172.21.15.156:3300/0 conn(0x7f82b4143fd0 0x7f82b40ceba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b37fe640 1 -- 172.21.15.16:0/1939951366 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1013349267 0 0) 0x7f82b40ce410 con 0x7f82b40cf0e0 2024-04-24T23:49:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b37fe640 1 -- 172.21.15.16:0/1939951366 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82a4003520 con 0x7f82b40cf0e0 2024-04-24T23:49:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b37fe640 1 -- 172.21.15.16:0/1939951366 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1945395703 0 0) 0x7f82a4003520 con 0x7f82b40cf0e0 2024-04-24T23:49:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b37fe640 1 -- 172.21.15.16:0/1939951366 >> v2:172.21.15.156:3300/0 conn(0x7f82b4143fd0 msgr2=0x7f82b40ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b37fe640 1 --2- 172.21.15.16:0/1939951366 >> v2:172.21.15.156:3300/0 conn(0x7f82b4143fd0 0x7f82b40ceba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b37fe640 1 -- 172.21.15.16:0/1939951366 >> v1:172.21.15.156:6789/0 conn(0x7f82b4140bb0 legacy=0x7f82b415e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b37fe640 1 -- 172.21.15.16:0/1939951366 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82b415ee80 con 0x7f82b40cf0e0 2024-04-24T23:49:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.683+0000 7f82b37fe640 1 -- 172.21.15.16:0/1939951366 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f82ac0019f0 con 0x7f82b40cf0e0 2024-04-24T23:49:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b37fe640 1 -- 172.21.15.16:0/1939951366 >> v1:172.21.15.16:6789/0 conn(0x7f82b40cf0e0 legacy=0x7f82b40d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b37fe640 1 --2- 172.21.15.16:0/1939951366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82a40039b0 0x7f82a4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b37fe640 1 --2- 172.21.15.16:0/1939951366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a4004800 0x7f82a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b37fe640 1 --2- 172.21.15.16:0/1939951366 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82a40051c0 0x7f82a400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b37fe640 1 -- 172.21.15.16:0/1939951366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82b415e980 con 0x7f82a4004800 2024-04-24T23:49:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b37fe640 1 -- 172.21.15.16:0/1939951366 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f82a400c060 con 0x7f82a40039b0 2024-04-24T23:49:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b37fe640 1 -- 172.21.15.16:0/1939951366 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f82a400c260 con 0x7f82a40051c0 2024-04-24T23:49:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b3fff640 1 --2- 172.21.15.16:0/1939951366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a4004800 0x7f82a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b899e640 1 --2- 172.21.15.16:0/1939951366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82a40039b0 0x7f82a4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b919f640 1 --2- 172.21.15.16:0/1939951366 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82a40051c0 0x7f82a400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b3fff640 1 -- 172.21.15.16:0/1939951366 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82a40051c0 msgr2=0x7f82a400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b3fff640 1 --2- 172.21.15.16:0/1939951366 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82a40051c0 0x7f82a400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b3fff640 1 -- 172.21.15.16:0/1939951366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82a40039b0 msgr2=0x7f82a4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b3fff640 1 --2- 172.21.15.16:0/1939951366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82a40039b0 0x7f82a4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b3fff640 1 -- 172.21.15.16:0/1939951366 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82a4003520 con 0x7f82a4004800 2024-04-24T23:49:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b899e640 1 --2- 172.21.15.16:0/1939951366 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82a40039b0 0x7f82a4003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.684+0000 7f82b919f640 1 -- 172.21.15.16:0/1939951366 reap_dead start 2024-04-24T23:49:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.685+0000 7f82b3fff640 1 --2- 172.21.15.16:0/1939951366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a4004800 0x7f82a4004c80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009ac0 tx=0x7f82a4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fac485ae8d3ac34d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.685+0000 7f82b37fe640 1 -- 172.21.15.16:0/1939951366 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a0036070 con 0x7f82a4004800 2024-04-24T23:49:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.685+0000 7f82b37fe640 1 -- 172.21.15.16:0/1939951366 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82a0031400 con 0x7f82a4004800 2024-04-24T23:49:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.685+0000 7f82b37fe640 1 -- 172.21.15.16:0/1939951366 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a0009690 con 0x7f82a4004800 2024-04-24T23:49:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.685+0000 7f82b99a0640 1 -- 172.21.15.16:0/1939951366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a4004800 msgr2=0x7f82a4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.685+0000 7f82b99a0640 1 --2- 172.21.15.16:0/1939951366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a4004800 0x7f82a4004c80 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009ac0 tx=0x7f82a4005830 comp rx=0 tx=0).stop 2024-04-24T23:49:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.685+0000 7f82b99a0640 1 -- 172.21.15.16:0/1939951366 shutdown_connections 2024-04-24T23:49:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.685+0000 7f82b99a0640 1 --2- 172.21.15.16:0/1939951366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82a4004800 0x7f82a4004c80 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.685+0000 7f82b99a0640 1 -- 172.21.15.16:0/1939951366 >> 172.21.15.16:0/1939951366 conn(0x7f82b406b2c0 msgr2=0x7f82b406b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.685+0000 7f82b99a0640 1 -- 172.21.15.16:0/1939951366 shutdown_connections 2024-04-24T23:49:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.685+0000 7f82b99a0640 1 -- 172.21.15.16:0/1939951366 wait complete. 2024-04-24T23:49:02.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.687+0000 7f82b99a0640 1 Processor -- start 2024-04-24T23:49:02.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.687+0000 7f82b99a0640 1 -- start start 2024-04-24T23:49:02.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.687+0000 7f82b99a0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82a4004800 0x7f82b4144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.687+0000 7f82b99a0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82b41f3cb0 0x7f82b41574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.687+0000 7f82b99a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b40b5930 0x7f82b40b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.687+0000 7f82b99a0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82b415f160 con 0x7f82b40b5930 2024-04-24T23:49:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.687+0000 7f82b99a0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f82b40ce710 con 0x7f82b41f3cb0 2024-04-24T23:49:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b99a0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f82b40ceba0 con 0x7f82a4004800 2024-04-24T23:49:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b919f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b40b5930 0x7f82b40b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b919f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b40b5930 0x7f82b40b5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43482/0 (socket says 172.21.15.16:43482) 2024-04-24T23:49:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b919f640 1 -- 172.21.15.16:0/2991598500 learned_addr learned my addr 172.21.15.16:0/2991598500 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b3fff640 1 --2- 172.21.15.16:0/2991598500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82b41f3cb0 0x7f82b41574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b899e640 1 --2- 172.21.15.16:0/2991598500 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82a4004800 0x7f82b4144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b919f640 1 -- 172.21.15.16:0/2991598500 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82a4004800 msgr2=0x7f82b4144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b919f640 1 --2- 172.21.15.16:0/2991598500 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82a4004800 0x7f82b4144cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b919f640 1 -- 172.21.15.16:0/2991598500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82b41f3cb0 msgr2=0x7f82b41574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b919f640 1 --2- 172.21.15.16:0/2991598500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82b41f3cb0 0x7f82b41574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b919f640 1 -- 172.21.15.16:0/2991598500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82b4145860 con 0x7f82b40b5930 2024-04-24T23:49:02.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b919f640 1 --2- 172.21.15.16:0/2991598500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b40b5930 0x7f82b40b5db0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f82b40b48b0 tx=0x7f82ac00e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b17fa640 1 -- 172.21.15.16:0/2991598500 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ac00ec30 con 0x7f82b40b5930 2024-04-24T23:49:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b17fa640 1 -- 172.21.15.16:0/2991598500 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82ac00edf0 con 0x7f82b40b5930 2024-04-24T23:49:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b99a0640 1 -- 172.21.15.16:0/2991598500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82b41452a0 con 0x7f82b40b5930 2024-04-24T23:49:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b17fa640 1 -- 172.21.15.16:0/2991598500 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ac0146c0 con 0x7f82b40b5930 2024-04-24T23:49:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.688+0000 7f82b99a0640 1 -- 172.21.15.16:0/2991598500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82b415a1f0 con 0x7f82b40b5930 2024-04-24T23:49:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.690+0000 7f82b17fa640 1 -- 172.21.15.16:0/2991598500 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f82ac014860 con 0x7f82b40b5930 2024-04-24T23:49:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.690+0000 7f82b17fa640 1 --2- 172.21.15.16:0/2991598500 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f829806be30 0x7f829806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.691+0000 7f82b17fa640 1 -- 172.21.15.16:0/2991598500 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f82ac08b040 con 0x7f82b40b5930 2024-04-24T23:49:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.691+0000 7f82b899e640 1 --2- 172.21.15.16:0/2991598500 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f829806be30 0x7f829806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.691+0000 7f82b899e640 1 --2- 172.21.15.16:0/2991598500 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f829806be30 0x7f829806e2f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f82a8000c00 tx=0x7f82a8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.692+0000 7f82bd9ea740 1 Processor -- start 2024-04-24T23:49:02.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.692+0000 7f82bd9ea740 1 -- start start 2024-04-24T23:49:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.692+0000 7f82bd9ea740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41bc4d60 0x558b41b776f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.692+0000 7f82bd9ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41ba7e80 0x558b41ba7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.692+0000 7f82bd9ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7740 0x558b41b7c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.692+0000 7f82bd9ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558b41adff00 con 0x558b41ba7e80 2024-04-24T23:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.692+0000 7f82bd9ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558b41a88750 con 0x558b41ba7740 2024-04-24T23:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.692+0000 7f82bd9ea740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x558b41adbec0 con 0x558b41bc4d60 2024-04-24T23:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.692+0000 7f82b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41ba7e80 0x558b41ba7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.692+0000 7f82b3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41ba7e80 0x558b41ba7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43486/0 (socket says 172.21.15.16:43486) 2024-04-24T23:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.692+0000 7f82b3fff640 1 -- 172.21.15.16:0/3562785650 learned_addr learned my addr 172.21.15.16:0/3562785650 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.692+0000 7f82b919f640 1 --2- 172.21.15.16:0/3562785650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7740 0x558b41b7c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.692+0000 7f82b899e640 1 --2- 172.21.15.16:0/3562785650 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41bc4d60 0x558b41b776f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.693+0000 7f82b3fff640 1 -- 172.21.15.16:0/3562785650 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41bc4d60 msgr2=0x558b41b776f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:02.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.693+0000 7f82b3fff640 1 --2- 172.21.15.16:0/3562785650 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41bc4d60 0x558b41b776f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.693+0000 7f82b3fff640 1 -- 172.21.15.16:0/3562785650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7740 msgr2=0x558b41b7c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.693+0000 7f82b3fff640 1 --2- 172.21.15.16:0/3562785650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7740 0x558b41b7c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.693+0000 7f82b3fff640 1 -- 172.21.15.16:0/3562785650 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b417f42d0 con 0x558b41ba7e80 2024-04-24T23:49:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.693+0000 7f82b3fff640 1 --2- 172.21.15.16:0/3562785650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41ba7e80 0x558b41ba7200 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009be0 tx=0x7f82a002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fa4dc4882616a37 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.693+0000 7f829e7fc640 1 -- 172.21.15.16:0/3562785650 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a0036040 con 0x558b41ba7e80 2024-04-24T23:49:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.693+0000 7f829e7fc640 1 -- 172.21.15.16:0/3562785650 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82a0037040 con 0x558b41ba7e80 2024-04-24T23:49:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.693+0000 7f829e7fc640 1 -- 172.21.15.16:0/3562785650 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a0003610 con 0x558b41ba7e80 2024-04-24T23:49:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.693+0000 7f82bd9ea740 1 -- 172.21.15.16:0/3562785650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41ba7e80 msgr2=0x558b41ba7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.693+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/3562785650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41ba7e80 0x558b41ba7200 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f82a0009be0 tx=0x7f82a002e000 comp rx=0 tx=0).stop 2024-04-24T23:49:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.694+0000 7f82bd9ea740 1 -- 172.21.15.16:0/3562785650 shutdown_connections 2024-04-24T23:49:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.694+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/3562785650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7740 0x558b41b7c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.694+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/3562785650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41ba7e80 0x558b41ba7200 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.694+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/3562785650 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41bc4d60 0x558b41b776f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.694+0000 7f82bd9ea740 1 -- 172.21.15.16:0/3562785650 >> 172.21.15.16:0/3562785650 conn(0x558b41b79960 msgr2=0x558b41bc2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.694+0000 7f82bd9ea740 1 -- 172.21.15.16:0/3562785650 shutdown_connections 2024-04-24T23:49:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.694+0000 7f82bd9ea740 1 -- 172.21.15.16:0/3562785650 wait complete. 2024-04-24T23:49:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.694+0000 7f82bd9ea740 1 -- start start 2024-04-24T23:49:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.694+0000 7f82bd9ea740 1 Processor -- start 2024-04-24T23:49:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.695+0000 7f82bd9ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7e80 0x558b41b97be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.695+0000 7f82bd9ea740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41bc4d60 0x558b41ba9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.695+0000 7f82bd9ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41baa3f0 0x558b41b7bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.695+0000 7f82bd9ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558b41b7bb10 con 0x558b41baa3f0 2024-04-24T23:49:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.695+0000 7f82bd9ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558b41a9bcf0 con 0x558b41ba7e80 2024-04-24T23:49:02.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.695+0000 7f82bd9ea740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x558b41b7af00 con 0x558b41bc4d60 2024-04-24T23:49:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.695+0000 7f82bd9ea740 10 client.0 _ll_register_callbacks cb 0x558b41a9b6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:49:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.695+0000 7f82b919f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41baa3f0 0x558b41b7bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.695+0000 7f82b919f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41baa3f0 0x558b41b7bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43498/0 (socket says 172.21.15.16:43498) 2024-04-24T23:49:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.695+0000 7f82b919f640 1 -- 172.21.15.16:0/4204063452 learned_addr learned my addr 172.21.15.16:0/4204063452 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.695+0000 7f82b3fff640 1 --2- 172.21.15.16:0/4204063452 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41bc4d60 0x558b41ba9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.695+0000 7f82b899e640 1 --2- 172.21.15.16:0/4204063452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7e80 0x558b41b97be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82b919f640 1 -- 172.21.15.16:0/4204063452 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41bc4d60 msgr2=0x558b41ba9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82b919f640 1 --2- 172.21.15.16:0/4204063452 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41bc4d60 0x558b41ba9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82b919f640 1 -- 172.21.15.16:0/4204063452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7e80 msgr2=0x558b41b97be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82b919f640 1 --2- 172.21.15.16:0/4204063452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7e80 0x558b41b97be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82b919f640 1 -- 172.21.15.16:0/4204063452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b418be6b0 con 0x558b41baa3f0 2024-04-24T23:49:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82b919f640 1 --2- 172.21.15.16:0/4204063452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41baa3f0 0x558b41b7bf10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f82ac007c40 tx=0x7f82ac059130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac367435a537b5cd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82937fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-24T23:49:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82937fe640 1 -- 172.21.15.16:0/4204063452 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ac088060 con 0x558b41baa3f0 2024-04-24T23:49:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82937fe640 1 -- 172.21.15.16:0/4204063452 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f82ac082b70 con 0x558b41baa3f0 2024-04-24T23:49:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82bd9ea740 1 -- 172.21.15.16:0/4204063452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82a0037040 con 0x558b41baa3f0 2024-04-24T23:49:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82937fe640 1 -- 172.21.15.16:0/4204063452 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7f82ac077050 con 0x558b41baa3f0 2024-04-24T23:49:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82bd9ea740 10 client.4407 fetch_fsmap learned FSMap version 6 2024-04-24T23:49:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82bd9ea740 1 -- 172.21.15.16:0/4204063452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x558b41b95fa0 con 0x558b41baa3f0 2024-04-24T23:49:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82937fe640 1 -- 172.21.15.16:0/4204063452 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82a0037040 con 0x558b41baa3f0 2024-04-24T23:49:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82937fe640 1 -- 172.21.15.16:0/4204063452 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7f82ac014b50 con 0x558b41baa3f0 2024-04-24T23:49:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82bd9ea740 10 client.4407 fetch_fsmap finished waiting for FSMap version 6 2024-04-24T23:49:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82bd9ea740 10 client.4407 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T23:49:02.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82bd9ea740 20 client.4407 populate_metadata read hostname 'smithi016' 2024-04-24T23:49:02.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/4204063452 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x558b41b953b0 0x558b41ba89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82bd9ea740 4 client.4407 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:49:02.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.696+0000 7f82bd9ea740 1 -- 172.21.15.16:0/4204063452 --> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b41aed440 con 0x558b41b953b0 2024-04-24T23:49:02.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.697+0000 7f82b899e640 1 --2- 172.21.15.16:0/4204063452 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x558b41b953b0 0x558b41ba89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.697+0000 7f82b899e640 1 --2- 172.21.15.16:0/4204063452 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x558b41b953b0 0x558b41ba89f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdee66c605846af5 server_cookie=1d5398a8c148fb96 in_seq=0 out_seq=0 2024-04-24T23:49:02.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.697+0000 7f82937fe640 10 client.4407 ms_handle_connect on v2:172.21.15.156:6835/289127714 2024-04-24T23:49:02.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82937fe640 1 -- 172.21.15.16:0/4204063452 <== mds.0 v2:172.21.15.156:6835/289127714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b41aed440 con 0x558b41b953b0 2024-04-24T23:49:02.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82937fe640 10 client.4407 handle_command_reply: tid=0 2024-04-24T23:49:02.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82bd9ea740 1 client.4407 shutdown 2024-04-24T23:49:02.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82bd9ea740 1 -- 172.21.15.16:0/4204063452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41baa3f0 msgr2=0x558b41b7bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/4204063452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41baa3f0 0x558b41b7bf10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f82ac007c40 tx=0x7f82ac059130 comp rx=0 tx=0).stop 2024-04-24T23:49:02.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82bd9ea740 1 -- 172.21.15.16:0/4204063452 shutdown_connections 2024-04-24T23:49:02.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/4204063452 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x558b41b953b0 0x558b41ba89f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/4204063452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41baa3f0 0x558b41b7bf10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/4204063452 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41bc4d60 0x558b41ba9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82937fe640 0 client.4407 ms_handle_reset on v2:172.21.15.156:6835/289127714 2024-04-24T23:49:02.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/4204063452 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7e80 0x558b41b97be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82bd9ea740 1 -- 172.21.15.16:0/4204063452 >> 172.21.15.16:0/4204063452 conn(0x558b41ba7740 msgr2=0x558b41b76e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:02.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82bd9ea740 1 -- 172.21.15.16:0/4204063452 shutdown_connections 2024-04-24T23:49:02.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82bd9ea740 1 -- 172.21.15.16:0/4204063452 wait complete. 2024-04-24T23:49:02.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.698+0000 7f82bd9ea740 20 client.4407 trim_cache size 0 max 16384 2024-04-24T23:49:02.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.716+0000 7f82bd9ea740 1 Processor -- start 2024-04-24T23:49:02.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.716+0000 7f82bd9ea740 1 -- start start 2024-04-24T23:49:02.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82bd9ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7e80 0x558b41be5d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82bd9ea740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41baa3f0 0x558b41ba8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82bd9ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41bc4d60 0x558b41ba8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82bd9ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82b4201100 con 0x558b41bc4d60 2024-04-24T23:49:02.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82bd9ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f82b4159d90 con 0x558b41ba7e80 2024-04-24T23:49:02.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82bd9ea740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f82b4200a90 con 0x558b41baa3f0 2024-04-24T23:49:02.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82b919f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41bc4d60 0x558b41ba8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82b919f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41bc4d60 0x558b41ba8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43506/0 (socket says 172.21.15.16:43506) 2024-04-24T23:49:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82b919f640 1 -- 172.21.15.16:0/2382761412 learned_addr learned my addr 172.21.15.16:0/2382761412 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82b919f640 1 -- 172.21.15.16:0/2382761412 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41baa3f0 msgr2=0x558b41ba8970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82b3fff640 1 --2- 172.21.15.16:0/2382761412 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41baa3f0 0x558b41ba8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82b899e640 1 --2- 172.21.15.16:0/2382761412 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7e80 0x558b41be5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82b919f640 1 --2- 172.21.15.16:0/2382761412 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41baa3f0 0x558b41ba8970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82b919f640 1 -- 172.21.15.16:0/2382761412 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7e80 msgr2=0x558b41be5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82b919f640 1 --2- 172.21.15.16:0/2382761412 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7e80 0x558b41be5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82b919f640 1 -- 172.21.15.16:0/2382761412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b41b784d0 con 0x558b41bc4d60 2024-04-24T23:49:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82b3fff640 1 --2- 172.21.15.16:0/2382761412 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41baa3f0 0x558b41ba8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:02.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82b899e640 1 --2- 172.21.15.16:0/2382761412 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7e80 0x558b41be5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.717+0000 7f82b919f640 1 --2- 172.21.15.16:0/2382761412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41bc4d60 0x558b41ba8eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f82ac058fd0 tx=0x7f82ac06e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1291e63f86ff618e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.718+0000 7f829dffb640 1 -- 172.21.15.16:0/2382761412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ac0589a0 con 0x558b41bc4d60 2024-04-24T23:49:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.718+0000 7f829dffb640 1 -- 172.21.15.16:0/2382761412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82ac014b50 con 0x558b41bc4d60 2024-04-24T23:49:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.718+0000 7f829dffb640 1 -- 172.21.15.16:0/2382761412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ac082db0 con 0x558b41bc4d60 2024-04-24T23:49:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.718+0000 7f82bd9ea740 1 -- 172.21.15.16:0/2382761412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41bc4d60 msgr2=0x558b41ba8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.718+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/2382761412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41bc4d60 0x558b41ba8eb0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f82ac058fd0 tx=0x7f82ac06e000 comp rx=0 tx=0).stop 2024-04-24T23:49:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.718+0000 7f82bd9ea740 1 -- 172.21.15.16:0/2382761412 shutdown_connections 2024-04-24T23:49:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.718+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/2382761412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41bc4d60 0x558b41ba8eb0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.718+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/2382761412 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41baa3f0 0x558b41ba8970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.718+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/2382761412 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41ba7e80 0x558b41be5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.718+0000 7f82bd9ea740 1 -- 172.21.15.16:0/2382761412 >> 172.21.15.16:0/2382761412 conn(0x558b41ba7740 msgr2=0x558b41bbfd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:02.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.718+0000 7f82bd9ea740 1 -- 172.21.15.16:0/2382761412 shutdown_connections 2024-04-24T23:49:02.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.718+0000 7f82bd9ea740 1 -- 172.21.15.16:0/2382761412 wait complete. 2024-04-24T23:49:02.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.719+0000 7f82bd9ea740 1 -- start start 2024-04-24T23:49:02.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.719+0000 7f82bd9ea740 1 Processor -- start 2024-04-24T23:49:02.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82bd9ea740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41ba7e80 0x558b41c21b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82bd9ea740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41baa3f0 0x558b41bc3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.726 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82bd9ea740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41bc4d60 0x558b41ba8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82bd9ea740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x558b41b95230 con 0x558b41ba7e80 2024-04-24T23:49:02.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82bd9ea740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x558b41bc1620 con 0x558b41bc4d60 2024-04-24T23:49:02.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82bd9ea740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x558b41b95880 con 0x558b41baa3f0 2024-04-24T23:49:02.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82bd9ea740 10 client.0 _ll_register_callbacks cb 0x7f82ac058ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:49:02.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82b899e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41ba7e80 0x558b41c21b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82b899e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41ba7e80 0x558b41c21b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43522/0 (socket says 172.21.15.16:43522) 2024-04-24T23:49:02.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82b899e640 1 -- 172.21.15.16:0/2349502768 learned_addr learned my addr 172.21.15.16:0/2349502768 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:02.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82b919f640 1 --2- 172.21.15.16:0/2349502768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41bc4d60 0x558b41ba8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82b3fff640 1 --2- 172.21.15.16:0/2349502768 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41baa3f0 0x558b41bc3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82b899e640 1 -- 172.21.15.16:0/2349502768 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41baa3f0 msgr2=0x558b41bc3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:02.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82b899e640 1 --2- 172.21.15.16:0/2349502768 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41baa3f0 0x558b41bc3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82b899e640 1 -- 172.21.15.16:0/2349502768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41bc4d60 msgr2=0x558b41ba8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:02.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82b899e640 1 --2- 172.21.15.16:0/2349502768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41bc4d60 0x558b41ba8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82b899e640 1 -- 172.21.15.16:0/2349502768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b41b96a70 con 0x558b41ba7e80 2024-04-24T23:49:02.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82b3fff640 1 --2- 172.21.15.16:0/2349502768 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41baa3f0 0x558b41bc3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:02.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82b899e640 1 --2- 172.21.15.16:0/2349502768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41ba7e80 0x558b41c21b20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f82a8007aa0 tx=0x7f82a8001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fa25a3bd91b7d81 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:02.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82937fe640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-24T23:49:02.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82937fe640 1 -- 172.21.15.16:0/2349502768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82a802f070 con 0x558b41ba7e80 2024-04-24T23:49:02.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82bd9ea740 1 -- 172.21.15.16:0/2349502768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f82ac014b50 con 0x558b41ba7e80 2024-04-24T23:49:02.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82937fe640 1 -- 172.21.15.16:0/2349502768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f82a8016d20 con 0x558b41ba7e80 2024-04-24T23:49:02.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.720+0000 7f82937fe640 1 -- 172.21.15.16:0/2349502768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7f82a803a050 con 0x558b41ba7e80 2024-04-24T23:49:02.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.721+0000 7f82bd9ea740 10 client.4413 fetch_fsmap learned FSMap version 6 2024-04-24T23:49:02.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.721+0000 7f82bd9ea740 1 -- 172.21.15.16:0/2349502768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x558b41bc4370 con 0x558b41ba7e80 2024-04-24T23:49:02.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.721+0000 7f82937fe640 1 -- 172.21.15.16:0/2349502768 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82ac014b50 con 0x558b41ba7e80 2024-04-24T23:49:02.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.721+0000 7f82937fe640 1 -- 172.21.15.16:0/2349502768 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 6) ==== 3502+0+0 (secure 0 0 0) 0x7f82a803d020 con 0x558b41ba7e80 2024-04-24T23:49:02.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.721+0000 7f82bd9ea740 10 client.4413 fetch_fsmap finished waiting for FSMap version 6 2024-04-24T23:49:02.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.721+0000 7f82bd9ea740 10 client.4413 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T23:49:02.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.721+0000 7f82bd9ea740 20 client.4413 populate_metadata read hostname 'smithi016' 2024-04-24T23:49:02.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.721+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/2349502768 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x558b41bc5530 0x558b41bc5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.721+0000 7f82bd9ea740 4 client.4413 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:49:02.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.721+0000 7f82bd9ea740 1 -- 172.21.15.16:0/2349502768 --> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] -- command(tid 0: {"prefix": "status"}) -- 0x558b41ae7790 con 0x558b41bc5530 2024-04-24T23:49:02.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.721+0000 7f82b3fff640 1 --2- 172.21.15.16:0/2349502768 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x558b41bc5530 0x558b41bc5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.721+0000 7f82b3fff640 1 --2- 172.21.15.16:0/2349502768 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x558b41bc5530 0x558b41bc5910 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7097265fe48b08c server_cookie=dc1914a874d6ecc in_seq=0 out_seq=0 2024-04-24T23:49:02.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.721+0000 7f82937fe640 10 client.4413 ms_handle_connect on v2:172.21.15.156:6835/289127714 2024-04-24T23:49:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82937fe640 1 -- 172.21.15.16:0/2349502768 <== mds.0 v2:172.21.15.156:6835/289127714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x558b41ae7790 con 0x558b41bc5530 2024-04-24T23:49:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82937fe640 10 client.4413 handle_command_reply: tid=0 2024-04-24T23:49:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82bd9ea740 1 client.4413 shutdown 2024-04-24T23:49:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82bd9ea740 1 -- 172.21.15.16:0/2349502768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41ba7e80 msgr2=0x558b41c21b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/2349502768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41ba7e80 0x558b41c21b20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f82a8007aa0 tx=0x7f82a8001190 comp rx=0 tx=0).stop 2024-04-24T23:49:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82bd9ea740 1 -- 172.21.15.16:0/2349502768 shutdown_connections 2024-04-24T23:49:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/2349502768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x558b41bc4d60 0x558b41ba8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/2349502768 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x558b41baa3f0 0x558b41bc3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/2349502768 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x558b41bc5530 0x558b41bc5910 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82bd9ea740 1 --2- 172.21.15.16:0/2349502768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x558b41ba7e80 0x558b41c21b20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82937fe640 0 client.4413 ms_handle_reset on v2:172.21.15.156:6835/289127714 2024-04-24T23:49:02.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82bd9ea740 1 -- 172.21.15.16:0/2349502768 >> 172.21.15.16:0/2349502768 conn(0x558b41ba7740 msgr2=0x558b41b7c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82bd9ea740 1 -- 172.21.15.16:0/2349502768 shutdown_connections 2024-04-24T23:49:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82bd9ea740 1 -- 172.21.15.16:0/2349502768 wait complete. 2024-04-24T23:49:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82bd9ea740 20 client.4413 trim_cache size 0 max 16384 2024-04-24T23:49:02.731 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:49:02.732 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "f8999c8c-ab57-4aba-8d53-08e19e6ee3d5", 2024-04-24T23:49:02.732 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-04-24T23:49:02.732 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4294, 2024-04-24T23:49:02.732 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-24T23:49:02.732 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-24T23:49:02.732 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-24T23:49:02.732 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 7.2327638409999997, 2024-04-24T23:49:02.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 6, 2024-04-24T23:49:02.732 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 25, 2024-04-24T23:49:02.733 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 24, 2024-04-24T23:49:02.733 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 14.080564634 2024-04-24T23:49:02.733 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:49:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82b99a0640 1 -- 172.21.15.16:0/2991598500 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f829806be30 msgr2=0x7f829806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82b99a0640 1 --2- 172.21.15.16:0/2991598500 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f829806be30 0x7f829806e2f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f82a8000c00 tx=0x7f82a8009040 comp rx=0 tx=0).stop 2024-04-24T23:49:02.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82b99a0640 1 -- 172.21.15.16:0/2991598500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b40b5930 msgr2=0x7f82b40b5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82b99a0640 1 --2- 172.21.15.16:0/2991598500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b40b5930 0x7f82b40b5db0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f82b40b48b0 tx=0x7f82ac00e160 comp rx=0 tx=0).stop 2024-04-24T23:49:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.722+0000 7f82b99a0640 1 -- 172.21.15.16:0/2991598500 shutdown_connections 2024-04-24T23:49:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.723+0000 7f82b99a0640 1 --2- 172.21.15.16:0/2991598500 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f829806be30 0x7f829806e2f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.723+0000 7f82b99a0640 1 --2- 172.21.15.16:0/2991598500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b40b5930 0x7f82b40b5db0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.723+0000 7f82b99a0640 1 --2- 172.21.15.16:0/2991598500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82b41f3cb0 0x7f82b41574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.723+0000 7f82b99a0640 1 --2- 172.21.15.16:0/2991598500 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82a4004800 0x7f82b4144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.723+0000 7f82b99a0640 1 -- 172.21.15.16:0/2991598500 >> 172.21.15.16:0/2991598500 conn(0x7f82b406b2c0 msgr2=0x7f82b40cae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.723+0000 7f82b99a0640 1 -- 172.21.15.16:0/2991598500 shutdown_connections 2024-04-24T23:49:02.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.723+0000 7f82b99a0640 1 -- 172.21.15.16:0/2991598500 wait complete. 2024-04-24T23:49:02.739 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T23:49:02.739 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:49:02.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T23:49:02.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.839+0000 7fdfdf502640 1 Processor -- start 2024-04-24T23:49:02.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.840+0000 7fdfdf502640 1 -- start start 2024-04-24T23:49:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.840+0000 7fdfdf502640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdfd815e020 0x7fdfd80cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.840+0000 7fdfdf502640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfd80b3390 con 0x7fdfd80d1fa0 2024-04-24T23:49:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.840+0000 7fdfdf502640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfd806bfc0 con 0x7fdfd80b2bb0 2024-04-24T23:49:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.840+0000 7fdfdf502640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fdfd806c2a0 con 0x7fdfd815e020 2024-04-24T23:49:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.840+0000 7fdfdda9f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdfd80d1fa0 0x7fdfd80d23b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44506/0 (socket says 172.21.15.16:44506) 2024-04-24T23:49:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.840+0000 7fdfdda9f640 1 -- 172.21.15.16:0/1559036422 learned_addr learned my addr 172.21.15.16:0/1559036422 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:02.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.841+0000 7fdfdd29e640 1 --2- 172.21.15.16:0/1559036422 >> v2:172.21.15.156:3300/0 conn(0x7fdfd815e020 0x7fdfd80cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.841+0000 7fdfd7fff640 1 -- 172.21.15.16:0/1559036422 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 148643565 0 0) 0x7fdfd80b3390 con 0x7fdfd80d1fa0 2024-04-24T23:49:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.841+0000 7fdfd7fff640 1 -- 172.21.15.16:0/1559036422 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdfc0003520 con 0x7fdfd80d1fa0 2024-04-24T23:49:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.841+0000 7fdfdd29e640 1 -- 172.21.15.16:0/1559036422 >> v1:172.21.15.156:6789/0 conn(0x7fdfd80b2bb0 legacy=0x7fdfd80cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.841+0000 7fdfdd29e640 1 -- 172.21.15.16:0/1559036422 >> v1:172.21.15.16:6789/0 conn(0x7fdfd80d1fa0 legacy=0x7fdfd80d23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.841+0000 7fdfdd29e640 1 -- 172.21.15.16:0/1559036422 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd806c580 con 0x7fdfd815e020 2024-04-24T23:49:02.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.842+0000 7fdfdd29e640 1 --2- 172.21.15.16:0/1559036422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfd815e020 0x7fdfd80cf270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdfd00097a0 tx=0x7fdfd002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1fedae075ac28712 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:02.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.842+0000 7fdfd7fff640 1 -- 172.21.15.16:0/1559036422 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0037070 con 0x7fdfd815e020 2024-04-24T23:49:02.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.842+0000 7fdfd7fff640 1 -- 172.21.15.16:0/1559036422 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfd0032440 con 0x7fdfd815e020 2024-04-24T23:49:02.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.842+0000 7fdfd7fff640 1 -- 172.21.15.16:0/1559036422 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd00327f0 con 0x7fdfd815e020 2024-04-24T23:49:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.842+0000 7fdfdf502640 1 -- 172.21.15.16:0/1559036422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfd815e020 msgr2=0x7fdfd80cf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.842+0000 7fdfdf502640 1 --2- 172.21.15.16:0/1559036422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfd815e020 0x7fdfd80cf270 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdfd00097a0 tx=0x7fdfd002a1a0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.842+0000 7fdfdf502640 1 -- 172.21.15.16:0/1559036422 shutdown_connections 2024-04-24T23:49:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.842+0000 7fdfdf502640 1 --2- 172.21.15.16:0/1559036422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfd815e020 0x7fdfd80cf270 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.842+0000 7fdfdf502640 1 -- 172.21.15.16:0/1559036422 >> 172.21.15.16:0/1559036422 conn(0x7fdfd80d3e10 msgr2=0x7fdfd80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.842+0000 7fdfdf502640 1 -- 172.21.15.16:0/1559036422 shutdown_connections 2024-04-24T23:49:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.842+0000 7fdfdf502640 1 -- 172.21.15.16:0/1559036422 wait complete. 2024-04-24T23:49:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.844+0000 7fdfdf502640 1 Processor -- start 2024-04-24T23:49:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.844+0000 7fdfdf502640 1 -- start start 2024-04-24T23:49:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.844+0000 7fdfdf502640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfd80b2bb0 0x7fdfd81535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.844+0000 7fdfdf502640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfd80d1fa0 0x7fdfd80cb560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.844+0000 7fdfdf502640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfd815e020 0x7fdfd80b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.844+0000 7fdfdf502640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd815e400 con 0x7fdfd80b2bb0 2024-04-24T23:49:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.844+0000 7fdfdf502640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd80d1a80 con 0x7fdfd80d1fa0 2024-04-24T23:49:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.844+0000 7fdfdf502640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fdfd80cfec0 con 0x7fdfd815e020 2024-04-24T23:49:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.844+0000 7fdfdd29e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfd80b2bb0 0x7fdfd81535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.844+0000 7fdfdd29e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfd80b2bb0 0x7fdfd81535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43530/0 (socket says 172.21.15.16:43530) 2024-04-24T23:49:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.844+0000 7fdfdd29e640 1 -- 172.21.15.16:0/2584488984 learned_addr learned my addr 172.21.15.16:0/2584488984 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfdca9d640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfd80d1fa0 0x7fdfd80cb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfdda9f640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfd815e020 0x7fdfd80b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfdd29e640 1 -- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfd815e020 msgr2=0x7fdfd80b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfdd29e640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfd815e020 0x7fdfd80b4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfdd29e640 1 -- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfd80d1fa0 msgr2=0x7fdfd80cb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfdd29e640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfd80d1fa0 0x7fdfd80cb560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfdd29e640 1 -- 172.21.15.16:0/2584488984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd8153ae0 con 0x7fdfd80b2bb0 2024-04-24T23:49:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfdca9d640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfd80d1fa0 0x7fdfd80cb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfdda9f640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfd815e020 0x7fdfd80b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfdd29e640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfd80b2bb0 0x7fdfd81535a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdfd00097a0 tx=0x7fdfd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfd5ffb640 1 -- 172.21.15.16:0/2584488984 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0037050 con 0x7fdfd80b2bb0 2024-04-24T23:49:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfd5ffb640 1 -- 172.21.15.16:0/2584488984 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfd0038040 con 0x7fdfd80b2bb0 2024-04-24T23:49:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfdf502640 1 -- 172.21.15.16:0/2584488984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd80cd610 con 0x7fdfd80b2bb0 2024-04-24T23:49:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfd5ffb640 1 -- 172.21.15.16:0/2584488984 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0004470 con 0x7fdfd80b2bb0 2024-04-24T23:49:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.845+0000 7fdfdf502640 1 -- 172.21.15.16:0/2584488984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd80cdbd0 con 0x7fdfd80b2bb0 2024-04-24T23:49:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.846+0000 7fdfdf502640 1 -- 172.21.15.16:0/2584488984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd806dfe0 con 0x7fdfd80b2bb0 2024-04-24T23:49:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.847+0000 7fdfd5ffb640 1 -- 172.21.15.16:0/2584488984 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdfd003c6d0 con 0x7fdfd80b2bb0 2024-04-24T23:49:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.847+0000 7fdfd5ffb640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdfc406be30 0x7fdfc406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.847+0000 7fdfd5ffb640 1 -- 172.21.15.16:0/2584488984 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdfd00abb80 con 0x7fdfd80b2bb0 2024-04-24T23:49:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.847+0000 7fdfdca9d640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdfc406be30 0x7fdfc406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:02.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.848+0000 7fdfdca9d640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdfc406be30 0x7fdfc406e2f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdfd80cb1e0 tx=0x7fdfcc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:02.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:02.852+0000 7fdfd5ffb640 1 -- 172.21.15.16:0/2584488984 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdfd007a280 con 0x7fdfd80b2bb0 2024-04-24T23:49:03.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03.071+0000 7fdfdf502640 1 -- 172.21.15.16:0/2584488984 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fdfd804f5e0 con 0x7fdfd80b2bb0 2024-04-24T23:49:03.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03.071+0000 7fdfd5ffb640 1 -- 172.21.15.16:0/2584488984 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+88 (secure 0 0 0) 0x7fdfd0079b80 con 0x7fdfd80b2bb0 2024-04-24T23:49:03.071 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-24T23:49:03.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03.073+0000 7fdfdf502640 1 -- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdfc406be30 msgr2=0x7fdfc406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:03.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03.073+0000 7fdfdf502640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdfc406be30 0x7fdfc406e2f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdfd80cb1e0 tx=0x7fdfcc009040 comp rx=0 tx=0).stop 2024-04-24T23:49:03.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03.073+0000 7fdfdf502640 1 -- 172.21.15.16:0/2584488984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfd80b2bb0 msgr2=0x7fdfd81535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03.073+0000 7fdfdf502640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfd80b2bb0 0x7fdfd81535a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdfd00097a0 tx=0x7fdfd00046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03.073+0000 7fdfdf502640 1 -- 172.21.15.16:0/2584488984 shutdown_connections 2024-04-24T23:49:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03.073+0000 7fdfdf502640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdfc406be30 0x7fdfc406e2f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03.073+0000 7fdfdf502640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfd815e020 0x7fdfd80b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03.073+0000 7fdfdf502640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfd80d1fa0 0x7fdfd80cb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03.073+0000 7fdfdf502640 1 --2- 172.21.15.16:0/2584488984 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfd80b2bb0 0x7fdfd81535a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03.073+0000 7fdfdf502640 1 -- 172.21.15.16:0/2584488984 >> 172.21.15.16:0/2584488984 conn(0x7fdfd80d3e10 msgr2=0x7fdfd80b35c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03.074+0000 7fdfdf502640 1 -- 172.21.15.16:0/2584488984 shutdown_connections 2024-04-24T23:49:03.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03.074+0000 7fdfdf502640 1 -- 172.21.15.16:0/2584488984 wait complete. 2024-04-24T23:49:03.085 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-24T23:49:03.085 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T23:49:03.085 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-04-24T23:49:03.086 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-24T23:49:03.086 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:49:03.086 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-24T23:49:03.086 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T23:49:03.086 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T23:49:03.086 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-24T23:49:03.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:03.086 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-04-24T23:49:03.102 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T23:49:03.102 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T23:49:03.102 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T23:49:03.102 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:03.102 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-04-24T23:49:03.102 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:03.102 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:03.103 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:03.103 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-04-24T23:49:03.103 INFO:teuthology.orchestra.run.smithi016.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T23:49:03.103 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:03.103 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-04-24T23:49:03.103 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-24T23:49:03.103 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42733sec preferred_lft 42733sec 2024-04-24T23:49:03.103 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-04-24T23:49:03.103 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:03.103 INFO:teuthology.orchestra.run.smithi016.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:03.103 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:03.103 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-04-24T23:49:03.103 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:03.103 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:03.104 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-04-24T23:49:03.104 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-24T23:49:03.105 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:03.105 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:49:03.105 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2024-04-24T23:49:03.105 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2024-04-24T23:49:03.105 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2024-04-24T23:49:03.105 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-24T23:49:03.105 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:49:03.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:03.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:03.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:03.290 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T23:49:03.314 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-04-24T23:49:03.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:03.316 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-04-24T23:49:03.383 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-04-24T23:49:03.383 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T23:49:03.383 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-24T23:49:03.383 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-24T23:49:03.383 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T23:49:03.384 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:03.384 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:49:03.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-24T23:49:03.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-24T23:49:03.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-24T23:49:03.384 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:49:03.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:03.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:03.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:03.674 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-24T23:49:03.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:03.689 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-04-24T23:49:03.746 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:03.746 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:49:03.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:49:03.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-24T23:49:03.746 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:49:03.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:03.865 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-24T23:49:03.865 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:03.865 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:49:03.865 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-24T23:49:03.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-24T23:49:03.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-24T23:49:03.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-24T23:49:03.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-24T23:49:03.866 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:49:03.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:03.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:03 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:03.991 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:03.991 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:49:03.991 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-04-24T23:49:03.991 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2024-04-24T23:49:03.991 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:49:04.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:04.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:04.067 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-24T23:49:04.067 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:49:04.067 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:49:04.083 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-24T23:49:04.083 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:49:04.083 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:49:04.138 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-24T23:49:04.204 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:49:04.261 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-24T23:49:04.261 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-24T23:49:04.261 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-24T23:49:04.261 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-24T23:49:04.261 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-24T23:49:04.261 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-24T23:49:04.261 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-24T23:49:04.262 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-24T23:49:04.262 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-24T23:49:04.262 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-24T23:49:04.262 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-24T23:49:04.262 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-24T23:49:04.262 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:49:04.262 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-24T23:49:04.262 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-24T23:49:04.262 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-24T23:49:04.262 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-24T23:49:04.262 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-24T23:49:04.262 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-04-24T23:49:04.262 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-24T23:49:04.263 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:49:04.263 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:49:04.263 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:49:04.263 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:49:04.263 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-24T23:49:04.263 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-24T23:49:04.263 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T23:49:04.263 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T23:49:04.263 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T23:49:04.263 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T23:49:04.263 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-24T23:49:04.263 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:49:04.263 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:49:04.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:04.264 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-24T23:49:04.317 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T23:49:04.317 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-24T23:49:04.360 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-24T23:49:04.384 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:49:04.440 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-24T23:49:04.441 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-24T23:49:04.441 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-24T23:49:04.441 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-24T23:49:04.441 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-24T23:49:04.441 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-24T23:49:04.441 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-24T23:49:04.441 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-24T23:49:04.441 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-24T23:49:04.441 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-24T23:49:04.442 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-24T23:49:04.442 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-24T23:49:04.442 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:49:04.442 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-24T23:49:04.442 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-24T23:49:04.442 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-24T23:49:04.442 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-24T23:49:04.442 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-24T23:49:04.442 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-04-24T23:49:04.442 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-24T23:49:04.442 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:49:04.442 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:49:04.442 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:49:04.442 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:49:04.443 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-24T23:49:04.443 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-24T23:49:04.443 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T23:49:04.443 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T23:49:04.443 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T23:49:04.443 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T23:49:04.443 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-24T23:49:04.443 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:49:04.443 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:49:04.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:04.444 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-24T23:49:04.448 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:2024-04-24T23:49:04.449+0000 7f0eb09c8480 -1 init, newargv = 0x55e867da8310 newargc=15 2024-04-24T23:49:04.449 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[41178]: starting ceph client 2024-04-24T23:49:04.454 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[41178]: starting fuse 2024-04-24T23:49:04.464 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-04-24T23:49:04.464 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-24T23:49:04.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-04-24T23:49:04.464 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-04-24T23:49:04.464 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-04-24T23:49:04.464 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-04-24T23:49:04.464 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-04-24T23:49:04.464 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:49:04.464 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-04-24T23:49:04.464 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-04-24T23:49:04.465 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-04-24T23:49:04.466 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T23:49:04.466 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:49:04.466 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.admin")) 2024-04-24T23:49:04.466 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-04-24T23:49:04.585 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.admin.41178.asok 2024-04-24T23:49:04.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:04.595 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.41178.asok 2024-04-24T23:49:04.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:04.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.41178.asok status 2024-04-24T23:49:04.732 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:49:04.732 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-04-24T23:49:04.732 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-24T23:49:04.732 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-24T23:49:04.732 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "admin", 2024-04-24T23:49:04.732 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-04-24T23:49:04.732 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-24T23:49:04.732 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "41178", 2024-04-24T23:49:04.732 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-04-24T23:49:04.732 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4425, 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 4425 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2896040485 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:49:04.733 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-04-24T23:49:04.734 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-24T23:49:04.734 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-04-24T23:49:04.734 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 2896040485 2024-04-24T23:49:04.734 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:49:04.734 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.4425 192.168.0.1:0/2896040485", 2024-04-24T23:49:04.734 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/2896040485", 2024-04-24T23:49:04.734 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-04-24T23:49:04.734 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 6, 2024-04-24T23:49:04.734 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 25, 2024-04-24T23:49:04.734 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-04-24T23:49:04.734 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-04-24T23:49:04.734 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-04-24T23:49:04.734 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:49:04.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:04.740 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-04-24T23:49:04.740 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:49:04.756 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-04-24T23:49:04.757 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:49:04.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:04.757 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:49:04.822 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-24T23:49:04.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T23:49:04.928 INFO:teuthology.orchestra.run.smithi016.stdout:# file: . 2024-04-24T23:49:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-24T23:49:04.929 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:04.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:04.935 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:49:04.935 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:49:04.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.092+0000 7f07144ee640 1 Processor -- start 2024-04-24T23:49:05.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.092+0000 7f07144ee640 1 -- start start 2024-04-24T23:49:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.093+0000 7f07144ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f070c15dfa0 0x7f070c0d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.093+0000 7f07144ee640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f070c0ce5f0 0x7f070c0cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.093+0000 7f07144ee640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f070c0d05a0 con 0x7f070c0b5f00 2024-04-24T23:49:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.093+0000 7f07144ee640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f070c0b7360 con 0x7f070c0ce5f0 2024-04-24T23:49:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.093+0000 7f07144ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f070c0cc640 con 0x7f070c15dfa0 2024-04-24T23:49:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.093+0000 7f0711a89640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f070c0ce5f0 0x7f070c0cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.093+0000 7f0711a89640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f070c0ce5f0 0x7f070c0cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43538/0 (socket says 172.21.15.16:43538) 2024-04-24T23:49:05.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.093+0000 7f0711a89640 1 -- 172.21.15.16:0/365037508 learned_addr learned my addr 172.21.15.16:0/365037508 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.093+0000 7f071228a640 1 --2- 172.21.15.16:0/365037508 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f070c15dfa0 0x7f070c0d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.093+0000 7f0711a89640 1 -- 172.21.15.16:0/365037508 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f070c15dfa0 msgr2=0x7f070c0d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.093+0000 7f0711a89640 1 --2- 172.21.15.16:0/365037508 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f070c15dfa0 0x7f070c0d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.093+0000 7f0711a89640 1 -- 172.21.15.16:0/365037508 >> v1:172.21.15.156:6789/0 conn(0x7f070c0b5f00 legacy=0x7f070c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.094+0000 7f0711a89640 1 -- 172.21.15.16:0/365037508 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f070c0d0880 con 0x7f070c0ce5f0 2024-04-24T23:49:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.094+0000 7f0711a89640 1 --2- 172.21.15.16:0/365037508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070c0ce5f0 0x7f070c0cea70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0700009870 tx=0x7f070002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652d08444a40dfae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.094+0000 7f0711288640 1 -- 172.21.15.16:0/365037508 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0700037070 con 0x7f070c0ce5f0 2024-04-24T23:49:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.094+0000 7f0711288640 1 -- 172.21.15.16:0/365037508 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0700032440 con 0x7f070c0ce5f0 2024-04-24T23:49:05.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.094+0000 7f0711288640 1 -- 172.21.15.16:0/365037508 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0700032810 con 0x7f070c0ce5f0 2024-04-24T23:49:05.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.095+0000 7f07144ee640 1 -- 172.21.15.16:0/365037508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070c0ce5f0 msgr2=0x7f070c0cea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.095+0000 7f07144ee640 1 --2- 172.21.15.16:0/365037508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070c0ce5f0 0x7f070c0cea70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0700009870 tx=0x7f070002a270 comp rx=0 tx=0).stop 2024-04-24T23:49:05.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.095+0000 7f07144ee640 1 -- 172.21.15.16:0/365037508 shutdown_connections 2024-04-24T23:49:05.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.095+0000 7f07144ee640 1 --2- 172.21.15.16:0/365037508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070c0ce5f0 0x7f070c0cea70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.095+0000 7f07144ee640 1 --2- 172.21.15.16:0/365037508 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f070c15dfa0 0x7f070c0d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.095+0000 7f07144ee640 1 -- 172.21.15.16:0/365037508 >> 172.21.15.16:0/365037508 conn(0x7f070c06b4d0 msgr2=0x7f070c0b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:05.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.095+0000 7f07144ee640 1 -- 172.21.15.16:0/365037508 shutdown_connections 2024-04-24T23:49:05.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.095+0000 7f07144ee640 1 -- 172.21.15.16:0/365037508 wait complete. 2024-04-24T23:49:05.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.096+0000 7f07144ee640 1 Processor -- start 2024-04-24T23:49:05.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.096+0000 7f07144ee640 1 -- start start 2024-04-24T23:49:05.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f07144ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f070c0b5f00 0x7f070c15b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f07144ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f070c0ce5f0 0x7f070c0afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f07144ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070c15dfa0 0x7f070c15f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f07144ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f070c15e380 con 0x7f070c15dfa0 2024-04-24T23:49:05.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f07144ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f070c0cc2d0 con 0x7f070c0ce5f0 2024-04-24T23:49:05.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f07144ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f070c0d1520 con 0x7f070c0b5f00 2024-04-24T23:49:05.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f0712a8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070c15dfa0 0x7f070c15f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f0712a8b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070c15dfa0 0x7f070c15f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43544/0 (socket says 172.21.15.16:43544) 2024-04-24T23:49:05.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f0712a8b640 1 -- 172.21.15.16:0/2769306443 learned_addr learned my addr 172.21.15.16:0/2769306443 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:05.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f071228a640 1 --2- 172.21.15.16:0/2769306443 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f070c0b5f00 0x7f070c15b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f0711a89640 1 --2- 172.21.15.16:0/2769306443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f070c0ce5f0 0x7f070c0afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f0712a8b640 1 -- 172.21.15.16:0/2769306443 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f070c0b5f00 msgr2=0x7f070c15b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f0712a8b640 1 --2- 172.21.15.16:0/2769306443 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f070c0b5f00 0x7f070c15b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f0712a8b640 1 -- 172.21.15.16:0/2769306443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f070c0ce5f0 msgr2=0x7f070c0afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f0712a8b640 1 --2- 172.21.15.16:0/2769306443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f070c0ce5f0 0x7f070c0afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.097+0000 7f0712a8b640 1 -- 172.21.15.16:0/2769306443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f070c1f7810 con 0x7f070c15dfa0 2024-04-24T23:49:05.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.098+0000 7f0712a8b640 1 --2- 172.21.15.16:0/2769306443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070c15dfa0 0x7f070c15f8c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f070400c400 tx=0x7f070400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:05.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.098+0000 7f070affd640 1 -- 172.21.15.16:0/2769306443 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0704017070 con 0x7f070c15dfa0 2024-04-24T23:49:05.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.098+0000 7f070affd640 1 -- 172.21.15.16:0/2769306443 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07040052d0 con 0x7f070c15dfa0 2024-04-24T23:49:05.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.098+0000 7f07144ee640 1 -- 172.21.15.16:0/2769306443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f070c1e9230 con 0x7f070c15dfa0 2024-04-24T23:49:05.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.098+0000 7f070affd640 1 -- 172.21.15.16:0/2769306443 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0704003810 con 0x7f070c15dfa0 2024-04-24T23:49:05.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.098+0000 7f07144ee640 1 -- 172.21.15.16:0/2769306443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f070c0af3e0 con 0x7f070c15dfa0 2024-04-24T23:49:05.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.099+0000 7f07144ee640 1 -- 172.21.15.16:0/2769306443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f070c06bea0 con 0x7f070c15dfa0 2024-04-24T23:49:05.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.099+0000 7f070affd640 1 -- 172.21.15.16:0/2769306443 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f070401a070 con 0x7f070c15dfa0 2024-04-24T23:49:05.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.100+0000 7f070affd640 1 --2- 172.21.15.16:0/2769306443 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f06f806be30 0x7f06f806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.100+0000 7f070affd640 1 -- 172.21.15.16:0/2769306443 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f070408a4c0 con 0x7f070c15dfa0 2024-04-24T23:49:05.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.100+0000 7f071228a640 1 --2- 172.21.15.16:0/2769306443 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f06f806be30 0x7f06f806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.100+0000 7f071228a640 1 --2- 172.21.15.16:0/2769306443 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f06f806be30 0x7f06f806e2f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f070c1f7450 tx=0x7f06fc009830 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:05.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.104+0000 7f070affd640 1 -- 172.21.15.16:0/2769306443 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f07040584e0 con 0x7f070c15dfa0 2024-04-24T23:49:05.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.289+0000 7f07144ee640 1 -- 172.21.15.16:0/2769306443 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f070c15d480 con 0x7f070c15dfa0 2024-04-24T23:49:05.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.290+0000 7f070affd640 1 -- 172.21.15.16:0/2769306443 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18502 (secure 0 0 0) 0x7f070405c7e0 con 0x7f070c15dfa0 2024-04-24T23:49:05.290 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:05.290 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:59.888015+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.292+0000 7f07144ee640 1 -- 172.21.15.16:0/2769306443 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f06f806be30 msgr2=0x7f06f806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.292+0000 7f07144ee640 1 --2- 172.21.15.16:0/2769306443 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f06f806be30 0x7f06f806e2f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f070c1f7450 tx=0x7f06fc009830 comp rx=0 tx=0).stop 2024-04-24T23:49:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.292+0000 7f07144ee640 1 -- 172.21.15.16:0/2769306443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070c15dfa0 msgr2=0x7f070c15f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.292+0000 7f07144ee640 1 --2- 172.21.15.16:0/2769306443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070c15dfa0 0x7f070c15f8c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f070400c400 tx=0x7f070400c8c0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.292+0000 7f07144ee640 1 -- 172.21.15.16:0/2769306443 shutdown_connections 2024-04-24T23:49:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.292+0000 7f07144ee640 1 --2- 172.21.15.16:0/2769306443 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f06f806be30 0x7f06f806e2f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.292+0000 7f07144ee640 1 --2- 172.21.15.16:0/2769306443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f070c15dfa0 0x7f070c15f8c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.292+0000 7f07144ee640 1 --2- 172.21.15.16:0/2769306443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f070c0ce5f0 0x7f070c0afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.292+0000 7f07144ee640 1 --2- 172.21.15.16:0/2769306443 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f070c0b5f00 0x7f070c15b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.292+0000 7f07144ee640 1 -- 172.21.15.16:0/2769306443 >> 172.21.15.16:0/2769306443 conn(0x7f070c06b4d0 msgr2=0x7f070c0b33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:05.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.292+0000 7f07144ee640 1 -- 172.21.15.16:0/2769306443 shutdown_connections 2024-04-24T23:49:05.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.292+0000 7f07144ee640 1 -- 172.21.15.16:0/2769306443 wait complete. 2024-04-24T23:49:05.303 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:05.303 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:05.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.407+0000 7f2abb13b640 1 Processor -- start 2024-04-24T23:49:05.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.407+0000 7f2abb13b640 1 -- start start 2024-04-24T23:49:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2abb13b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ab40d12d0 0x7f2ab40d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2abb13b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab415e5c0 0x7f2ab40d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2abb13b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ab415ec30 con 0x7f2ab40cd790 2024-04-24T23:49:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2abb13b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2ab415f4d0 con 0x7f2ab40d12d0 2024-04-24T23:49:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2abb13b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab415ef10 con 0x7f2ab415e5c0 2024-04-24T23:49:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2ab8ed7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ab40d12d0 0x7f2ab40d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2ab8ed7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ab40d12d0 0x7f2ab40d16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43560/0 (socket says 172.21.15.16:43560) 2024-04-24T23:49:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2ab8ed7640 1 -- 172.21.15.16:0/1225911331 learned_addr learned my addr 172.21.15.16:0/1225911331 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2ab3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2ab40cd790 0x7f2ab40c8580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44522/0 (socket says 172.21.15.16:44522) 2024-04-24T23:49:05.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2ab96d8640 1 --2- 172.21.15.16:0/1225911331 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab415e5c0 0x7f2ab40d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2ab8ed7640 1 -- 172.21.15.16:0/1225911331 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab415e5c0 msgr2=0x7f2ab40d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2ab8ed7640 1 --2- 172.21.15.16:0/1225911331 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab415e5c0 0x7f2ab40d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2ab8ed7640 1 -- 172.21.15.16:0/1225911331 >> v1:172.21.15.16:6789/0 conn(0x7f2ab40cd790 legacy=0x7f2ab40c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2ab8ed7640 1 -- 172.21.15.16:0/1225911331 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab415f1f0 con 0x7f2ab40d12d0 2024-04-24T23:49:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2ab37fe640 1 -- 172.21.15.16:0/1225911331 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1334372132 0 0) 0x7f2ab415ec30 con 0x7f2ab40cd790 2024-04-24T23:49:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.408+0000 7f2ab96d8640 1 --2- 172.21.15.16:0/1225911331 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab415e5c0 0x7f2ab40d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:05.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.409+0000 7f2ab8ed7640 1 --2- 172.21.15.16:0/1225911331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab40d12d0 0x7f2ab40d16b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ab415f650 tx=0x7f2aa802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0a541154015e52a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.409+0000 7f2ab37fe640 1 -- 172.21.15.16:0/1225911331 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8037070 con 0x7f2ab40d12d0 2024-04-24T23:49:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.409+0000 7f2ab37fe640 1 -- 172.21.15.16:0/1225911331 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aa8032440 con 0x7f2ab40d12d0 2024-04-24T23:49:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.409+0000 7f2ab37fe640 1 -- 172.21.15.16:0/1225911331 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8032810 con 0x7f2ab40d12d0 2024-04-24T23:49:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.410+0000 7f2abb13b640 1 -- 172.21.15.16:0/1225911331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab40d12d0 msgr2=0x7f2ab40d16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.410+0000 7f2abb13b640 1 --2- 172.21.15.16:0/1225911331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab40d12d0 0x7f2ab40d16b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2ab415f650 tx=0x7f2aa802a130 comp rx=0 tx=0).stop 2024-04-24T23:49:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.410+0000 7f2abb13b640 1 -- 172.21.15.16:0/1225911331 shutdown_connections 2024-04-24T23:49:05.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.410+0000 7f2abb13b640 1 --2- 172.21.15.16:0/1225911331 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab415e5c0 0x7f2ab40d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.410+0000 7f2abb13b640 1 --2- 172.21.15.16:0/1225911331 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab40d12d0 0x7f2ab40d16b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.410+0000 7f2abb13b640 1 -- 172.21.15.16:0/1225911331 >> 172.21.15.16:0/1225911331 conn(0x7f2ab4072e70 msgr2=0x7f2ab4073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:05.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.410+0000 7f2abb13b640 1 -- 172.21.15.16:0/1225911331 shutdown_connections 2024-04-24T23:49:05.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.410+0000 7f2abb13b640 1 -- 172.21.15.16:0/1225911331 wait complete. 2024-04-24T23:49:05.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.411+0000 7f2abb13b640 1 Processor -- start 2024-04-24T23:49:05.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.411+0000 7f2abb13b640 1 -- start start 2024-04-24T23:49:05.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2abb13b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab40cd790 0x7f2ab41574a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2abb13b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab40d12d0 0x7f2ab4153c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2abb13b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab415e5c0 0x7f2ab40caec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2abb13b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab40d27f0 con 0x7f2ab415e5c0 2024-04-24T23:49:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2abb13b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab40d2670 con 0x7f2ab40cd790 2024-04-24T23:49:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2abb13b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab40d2970 con 0x7f2ab40d12d0 2024-04-24T23:49:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2ab96d8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab415e5c0 0x7f2ab40caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2ab96d8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab415e5c0 0x7f2ab40caec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43576/0 (socket says 172.21.15.16:43576) 2024-04-24T23:49:05.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2ab96d8640 1 -- 172.21.15.16:0/3049271812 learned_addr learned my addr 172.21.15.16:0/3049271812 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2ab3fff640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab40d12d0 0x7f2ab4153c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2ab8ed7640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab40cd790 0x7f2ab41574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2ab96d8640 1 -- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab40d12d0 msgr2=0x7f2ab4153c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2ab96d8640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab40d12d0 0x7f2ab4153c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2ab96d8640 1 -- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab40cd790 msgr2=0x7f2ab41574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2ab96d8640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab40cd790 0x7f2ab41574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2ab96d8640 1 -- 172.21.15.16:0/3049271812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab4155930 con 0x7f2ab415e5c0 2024-04-24T23:49:05.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2ab8ed7640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab40cd790 0x7f2ab41574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:05.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2ab3fff640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab40d12d0 0x7f2ab4153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:05.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.412+0000 7f2ab96d8640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab415e5c0 0x7f2ab40caec0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00cef0 tx=0x7f2aac00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:05.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.413+0000 7f2ab17fa640 1 -- 172.21.15.16:0/3049271812 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac01a070 con 0x7f2ab415e5c0 2024-04-24T23:49:05.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.413+0000 7f2ab17fa640 1 -- 172.21.15.16:0/3049271812 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aac00d860 con 0x7f2ab415e5c0 2024-04-24T23:49:05.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.413+0000 7f2abb13b640 1 -- 172.21.15.16:0/3049271812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab4155c10 con 0x7f2ab415e5c0 2024-04-24T23:49:05.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.413+0000 7f2ab17fa640 1 -- 172.21.15.16:0/3049271812 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac00dc50 con 0x7f2ab415e5c0 2024-04-24T23:49:05.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.413+0000 7f2abb13b640 1 -- 172.21.15.16:0/3049271812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab41561d0 con 0x7f2ab415e5c0 2024-04-24T23:49:05.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.414+0000 7f2abb13b640 1 -- 172.21.15.16:0/3049271812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab4068f90 con 0x7f2ab415e5c0 2024-04-24T23:49:05.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.414+0000 7f2ab17fa640 1 -- 172.21.15.16:0/3049271812 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2aac01d070 con 0x7f2ab415e5c0 2024-04-24T23:49:05.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.415+0000 7f2ab17fa640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2a9806be30 0x7f2a9806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.415+0000 7f2ab17fa640 1 -- 172.21.15.16:0/3049271812 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f2aac08b9f0 con 0x7f2ab415e5c0 2024-04-24T23:49:05.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.415+0000 7f2ab8ed7640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2a9806be30 0x7f2a9806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.415+0000 7f2ab8ed7640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2a9806be30 0x7f2a9806e2f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8034000 tx=0x7f2aa802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:05.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.419+0000 7f2ab17fa640 1 -- 172.21.15.16:0/3049271812 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2aac05a1b0 con 0x7f2ab415e5c0 2024-04-24T23:49:05.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.603+0000 7f2abb13b640 1 -- 172.21.15.16:0/3049271812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2ab40c70e0 con 0x7f2ab415e5c0 2024-04-24T23:49:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.603+0000 7f2ab17fa640 1 -- 172.21.15.16:0/3049271812 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f2aac059ab0 con 0x7f2ab415e5c0 2024-04-24T23:49:05.603 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.605+0000 7f2abb13b640 1 -- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2a9806be30 msgr2=0x7f2a9806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.605+0000 7f2abb13b640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2a9806be30 0x7f2a9806e2f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8034000 tx=0x7f2aa802e000 comp rx=0 tx=0).stop 2024-04-24T23:49:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.605+0000 7f2abb13b640 1 -- 172.21.15.16:0/3049271812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab415e5c0 msgr2=0x7f2ab40caec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.605+0000 7f2abb13b640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab415e5c0 0x7f2ab40caec0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f2aac00cef0 tx=0x7f2aac00a710 comp rx=0 tx=0).stop 2024-04-24T23:49:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.605+0000 7f2abb13b640 1 -- 172.21.15.16:0/3049271812 shutdown_connections 2024-04-24T23:49:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.605+0000 7f2abb13b640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2a9806be30 0x7f2a9806e2f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.605+0000 7f2abb13b640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab415e5c0 0x7f2ab40caec0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.605+0000 7f2abb13b640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab40d12d0 0x7f2ab4153c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.605+0000 7f2abb13b640 1 --2- 172.21.15.16:0/3049271812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab40cd790 0x7f2ab41574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.605+0000 7f2abb13b640 1 -- 172.21.15.16:0/3049271812 >> 172.21.15.16:0/3049271812 conn(0x7f2ab4072e70 msgr2=0x7f2ab40b7460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.605+0000 7f2abb13b640 1 -- 172.21.15.16:0/3049271812 shutdown_connections 2024-04-24T23:49:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.605+0000 7f2abb13b640 1 -- 172.21.15.16:0/3049271812 wait complete. 2024-04-24T23:49:05.616 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.720+0000 7f15322a5640 1 Processor -- start 2024-04-24T23:49:05.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.720+0000 7f15322a5640 1 -- start start 2024-04-24T23:49:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.721+0000 7f15322a5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f152c06c0d0 0x7f152c0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.721+0000 7f15322a5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f152c0d24d0 con 0x7f152c06bc40 2024-04-24T23:49:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.721+0000 7f15322a5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f152c0d27b0 con 0x7f152c15dfd0 2024-04-24T23:49:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.721+0000 7f15322a5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f152c06ba00 con 0x7f152c06c0d0 2024-04-24T23:49:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.721+0000 7f152b7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f152c06bc40 0x7f152c0b7050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44532/0 (socket says 172.21.15.16:44532) 2024-04-24T23:49:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.721+0000 7f152b7fe640 1 -- 172.21.15.16:0/3064367834 learned_addr learned my addr 172.21.15.16:0/3064367834 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f1530842640 1 --2- 172.21.15.16:0/3064367834 >> v2:172.21.15.156:3300/0 conn(0x7f152c06c0d0 0x7f152c0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 -- 172.21.15.16:0/3064367834 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1486746927 0 0) 0x7f152c0d24d0 con 0x7f152c06bc40 2024-04-24T23:49:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 -- 172.21.15.16:0/3064367834 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1514003520 con 0x7f152c06bc40 2024-04-24T23:49:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 -- 172.21.15.16:0/3064367834 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3833044533 0 0) 0x7f1514003520 con 0x7f152c06bc40 2024-04-24T23:49:05.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 -- 172.21.15.16:0/3064367834 >> v2:172.21.15.156:3300/0 conn(0x7f152c06c0d0 msgr2=0x7f152c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 --2- 172.21.15.16:0/3064367834 >> v2:172.21.15.156:3300/0 conn(0x7f152c06c0d0 0x7f152c0b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 -- 172.21.15.16:0/3064367834 >> v1:172.21.15.156:6789/0 conn(0x7f152c15dfd0 legacy=0x7f152c06b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 -- 172.21.15.16:0/3064367834 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f152c0d2a90 con 0x7f152c06bc40 2024-04-24T23:49:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 -- 172.21.15.16:0/3064367834 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f1520001670 con 0x7f152c06bc40 2024-04-24T23:49:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 -- 172.21.15.16:0/3064367834 >> v1:172.21.15.16:6789/0 conn(0x7f152c06bc40 legacy=0x7f152c0b7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 --2- 172.21.15.16:0/3064367834 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15140039b0 0x7f1514003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 --2- 172.21.15.16:0/3064367834 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1514004800 0x7f1514004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 --2- 172.21.15.16:0/3064367834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15140051c0 0x7f151400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 -- 172.21.15.16:0/3064367834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f152c06ba00 con 0x7f15140051c0 2024-04-24T23:49:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 -- 172.21.15.16:0/3064367834 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f151400c160 con 0x7f15140039b0 2024-04-24T23:49:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152affd640 1 -- 172.21.15.16:0/3064367834 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f151400c360 con 0x7f1514004800 2024-04-24T23:49:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f1530842640 1 --2- 172.21.15.16:0/3064367834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15140051c0 0x7f151400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152bfff640 1 --2- 172.21.15.16:0/3064367834 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15140039b0 0x7f1514003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f152b7fe640 1 --2- 172.21.15.16:0/3064367834 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1514004800 0x7f1514004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f1530842640 1 -- 172.21.15.16:0/3064367834 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1514004800 msgr2=0x7f1514004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f1530842640 1 --2- 172.21.15.16:0/3064367834 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1514004800 0x7f1514004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f1530842640 1 -- 172.21.15.16:0/3064367834 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15140039b0 msgr2=0x7f1514003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f1530842640 1 --2- 172.21.15.16:0/3064367834 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15140039b0 0x7f1514003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.722+0000 7f1530842640 1 -- 172.21.15.16:0/3064367834 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1514003520 con 0x7f15140051c0 2024-04-24T23:49:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.723+0000 7f1530842640 1 -- 172.21.15.16:0/3064367834 reap_dead start 2024-04-24T23:49:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.723+0000 7f1530842640 1 --2- 172.21.15.16:0/3064367834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15140051c0 0x7f151400b9c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1524009fc0 tx=0x7f1514005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a2e87c2011fffb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.723+0000 7f152affd640 1 -- 172.21.15.16:0/3064367834 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524035020 con 0x7f15140051c0 2024-04-24T23:49:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.723+0000 7f152affd640 1 -- 172.21.15.16:0/3064367834 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1524003bd0 con 0x7f15140051c0 2024-04-24T23:49:05.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.723+0000 7f152affd640 1 -- 172.21.15.16:0/3064367834 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1524009c10 con 0x7f15140051c0 2024-04-24T23:49:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.724+0000 7f15322a5640 1 -- 172.21.15.16:0/3064367834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15140051c0 msgr2=0x7f151400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.724+0000 7f15322a5640 1 --2- 172.21.15.16:0/3064367834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15140051c0 0x7f151400b9c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1524009fc0 tx=0x7f1514005830 comp rx=0 tx=0).stop 2024-04-24T23:49:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.724+0000 7f15322a5640 1 -- 172.21.15.16:0/3064367834 shutdown_connections 2024-04-24T23:49:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.724+0000 7f15322a5640 1 --2- 172.21.15.16:0/3064367834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15140051c0 0x7f151400b9c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.724+0000 7f15322a5640 1 -- 172.21.15.16:0/3064367834 >> 172.21.15.16:0/3064367834 conn(0x7f152c0d3dc0 msgr2=0x7f152c0d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.724+0000 7f15322a5640 1 -- 172.21.15.16:0/3064367834 shutdown_connections 2024-04-24T23:49:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.724+0000 7f15322a5640 1 -- 172.21.15.16:0/3064367834 wait complete. 2024-04-24T23:49:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.725+0000 7f15322a5640 1 Processor -- start 2024-04-24T23:49:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.725+0000 7f15322a5640 1 -- start start 2024-04-24T23:49:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f15322a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15140051c0 0x7f152c149a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f15322a5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f152c158da0 0x7f152c0ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f15322a5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f152c0ca5a0 0x7f152c1603c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f15322a5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f152c06c4e0 con 0x7f15140051c0 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f15322a5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f152c0d3050 con 0x7f152c0ca5a0 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f15322a5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f152c0cead0 con 0x7f152c158da0 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f152bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15140051c0 0x7f152c149a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f152bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15140051c0 0x7f152c149a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43600/0 (socket says 172.21.15.16:43600) 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f152bfff640 1 -- 172.21.15.16:0/428608239 learned_addr learned my addr 172.21.15.16:0/428608239 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f152bfff640 1 -- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f152c158da0 msgr2=0x7f152c0ca060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f1530842640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f152c0ca5a0 0x7f152c1603c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f152b7fe640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f152c158da0 0x7f152c0ca060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f152bfff640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f152c158da0 0x7f152c0ca060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f152bfff640 1 -- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f152c0ca5a0 msgr2=0x7f152c1603c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f152bfff640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f152c0ca5a0 0x7f152c1603c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f152bfff640 1 -- 172.21.15.16:0/428608239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f152c147530 con 0x7f15140051c0 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f1530842640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f152c0ca5a0 0x7f152c1603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.726+0000 7f152bfff640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15140051c0 0x7f152c149a90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f151c00f5f0 tx=0x7f151c00fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:05.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.727+0000 7f1528ff9640 1 -- 172.21.15.16:0/428608239 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151c0052e0 con 0x7f15140051c0 2024-04-24T23:49:05.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.727+0000 7f1528ff9640 1 -- 172.21.15.16:0/428608239 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f151c018070 con 0x7f15140051c0 2024-04-24T23:49:05.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.727+0000 7f15322a5640 1 -- 172.21.15.16:0/428608239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f152c147810 con 0x7f15140051c0 2024-04-24T23:49:05.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.727+0000 7f1528ff9640 1 -- 172.21.15.16:0/428608239 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f151c003810 con 0x7f15140051c0 2024-04-24T23:49:05.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.727+0000 7f15322a5640 1 -- 172.21.15.16:0/428608239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f152c1480b0 con 0x7f15140051c0 2024-04-24T23:49:05.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.728+0000 7f15322a5640 1 -- 172.21.15.16:0/428608239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f152c147c00 con 0x7f15140051c0 2024-04-24T23:49:05.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.728+0000 7f1528ff9640 1 -- 172.21.15.16:0/428608239 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f151c0039b0 con 0x7f15140051c0 2024-04-24T23:49:05.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.729+0000 7f1528ff9640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f151806be60 0x7f151806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:05.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.729+0000 7f1528ff9640 1 -- 172.21.15.16:0/428608239 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f151c08afd0 con 0x7f15140051c0 2024-04-24T23:49:05.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.729+0000 7f152b7fe640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f151806be60 0x7f151806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:05.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.729+0000 7f152b7fe640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f151806be60 0x7f151806e320 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1520001050 tx=0x7f1520009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:05.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.733+0000 7f1528ff9640 1 -- 172.21.15.16:0/428608239 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f151c059640 con 0x7f15140051c0 2024-04-24T23:49:05.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.917+0000 7f15322a5640 1 -- 172.21.15.16:0/428608239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f152c068f90 con 0x7f15140051c0 2024-04-24T23:49:05.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.918+0000 7f1528ff9640 1 -- 172.21.15.16:0/428608239 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f151c058f40 con 0x7f15140051c0 2024-04-24T23:49:05.918 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:05.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.919+0000 7f15322a5640 1 -- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f151806be60 msgr2=0x7f151806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.919+0000 7f15322a5640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f151806be60 0x7f151806e320 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1520001050 tx=0x7f1520009040 comp rx=0 tx=0).stop 2024-04-24T23:49:05.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.919+0000 7f15322a5640 1 -- 172.21.15.16:0/428608239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15140051c0 msgr2=0x7f152c149a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:05.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.919+0000 7f15322a5640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15140051c0 0x7f152c149a90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f151c00f5f0 tx=0x7f151c00fab0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.920+0000 7f15322a5640 1 -- 172.21.15.16:0/428608239 shutdown_connections 2024-04-24T23:49:05.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.920+0000 7f15322a5640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f151806be60 0x7f151806e320 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.920+0000 7f15322a5640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f152c0ca5a0 0x7f152c1603c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.920+0000 7f15322a5640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f152c158da0 0x7f152c0ca060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.920+0000 7f15322a5640 1 --2- 172.21.15.16:0/428608239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15140051c0 0x7f152c149a90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:05.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.920+0000 7f15322a5640 1 -- 172.21.15.16:0/428608239 >> 172.21.15.16:0/428608239 conn(0x7f152c0d3dc0 msgr2=0x7f152c0b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:05.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.920+0000 7f15322a5640 1 -- 172.21.15.16:0/428608239 shutdown_connections 2024-04-24T23:49:05.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:05.920+0000 7f15322a5640 1 -- 172.21.15.16:0/428608239 wait complete. 2024-04-24T23:49:05.930 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:06.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.034+0000 7f226d053640 1 Processor -- start 2024-04-24T23:49:06.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.034+0000 7f226d053640 1 -- start start 2024-04-24T23:49:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226d053640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22680cd790 0x7f22680d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226d053640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f226815ec30 con 0x7f22680d12d0 2024-04-24T23:49:06.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226d053640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f226815ef10 con 0x7f226815e5c0 2024-04-24T23:49:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226d053640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f226815f4d0 con 0x7f22680cd790 2024-04-24T23:49:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f2266d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f22680d12d0 0x7f22680c8580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44534/0 (socket says 172.21.15.16:44534) 2024-04-24T23:49:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f2266d9d640 1 -- 172.21.15.16:0/1236239176 learned_addr learned my addr 172.21.15.16:0/1236239176 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f225ffff640 1 --2- 172.21.15.16:0/1236239176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22680cd790 0x7f22680d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4039100614 0 0) 0x7f226815ec30 con 0x7f22680d12d0 2024-04-24T23:49:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2254003520 con 0x7f22680d12d0 2024-04-24T23:49:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3926981864 0 0) 0x7f226815ef10 con 0x7f226815e5c0 2024-04-24T23:49:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f226815ec30 con 0x7f226815e5c0 2024-04-24T23:49:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4236369600 0 0) 0x7f2254003520 con 0x7f22680d12d0 2024-04-24T23:49:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22680cd790 msgr2=0x7f22680d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 --2- 172.21.15.16:0/1236239176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22680cd790 0x7f22680d16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 >> v1:172.21.15.156:6789/0 conn(0x7f226815e5c0 legacy=0x7f22680d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f226815f1f0 con 0x7f22680d12d0 2024-04-24T23:49:06.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f2258001670 con 0x7f22680d12d0 2024-04-24T23:49:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 >> v1:172.21.15.16:6789/0 conn(0x7f22680d12d0 legacy=0x7f22680c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 --2- 172.21.15.16:0/1236239176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22680cd790 0x7f2254003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 --2- 172.21.15.16:0/1236239176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22540040d0 0x7f2254004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 --2- 172.21.15.16:0/1236239176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2254004d40 0x7f22540051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f226815f4d0 con 0x7f2254004d40 2024-04-24T23:49:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f225400bbd0 con 0x7f22540040d0 2024-04-24T23:49:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.035+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f225400bdd0 con 0x7f22680cd790 2024-04-24T23:49:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.036+0000 7f226759e640 1 --2- 172.21.15.16:0/1236239176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2254004d40 0x7f22540051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.036+0000 7f2266d9d640 1 --2- 172.21.15.16:0/1236239176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22680cd790 0x7f2254003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.036+0000 7f225ffff640 1 --2- 172.21.15.16:0/1236239176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22540040d0 0x7f2254004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.036+0000 7f226759e640 1 -- 172.21.15.16:0/1236239176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22680cd790 msgr2=0x7f2254003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.036+0000 7f226759e640 1 --2- 172.21.15.16:0/1236239176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22680cd790 0x7f2254003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.036+0000 7f226759e640 1 -- 172.21.15.16:0/1236239176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22540040d0 msgr2=0x7f2254004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:06.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.036+0000 7f226759e640 1 --2- 172.21.15.16:0/1236239176 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22540040d0 0x7f2254004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.036+0000 7f226759e640 1 -- 172.21.15.16:0/1236239176 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2254003520 con 0x7f2254004d40 2024-04-24T23:49:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.036+0000 7f2266d9d640 1 --2- 172.21.15.16:0/1236239176 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22680cd790 0x7f2254003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.037+0000 7f226759e640 1 --2- 172.21.15.16:0/1236239176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2254004d40 0x7f22540051c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2260001fd0 tx=0x7f2254005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68ca75afe4f5dde server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.037+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2260001bf0 con 0x7f2254004d40 2024-04-24T23:49:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.037+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2260003b70 con 0x7f2254004d40 2024-04-24T23:49:06.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.037+0000 7f226659c640 1 -- 172.21.15.16:0/1236239176 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22600050d0 con 0x7f2254004d40 2024-04-24T23:49:06.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.037+0000 7f226d053640 1 -- 172.21.15.16:0/1236239176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2254004d40 msgr2=0x7f22540051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:06.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.037+0000 7f226d053640 1 --2- 172.21.15.16:0/1236239176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2254004d40 0x7f22540051c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2260001fd0 tx=0x7f2254005830 comp rx=0 tx=0).stop 2024-04-24T23:49:06.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.037+0000 7f226759e640 1 -- 172.21.15.16:0/1236239176 reap_dead start 2024-04-24T23:49:06.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.037+0000 7f226d053640 1 -- 172.21.15.16:0/1236239176 shutdown_connections 2024-04-24T23:49:06.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.037+0000 7f226d053640 1 -- 172.21.15.16:0/1236239176 >> 172.21.15.16:0/1236239176 conn(0x7f2268072e70 msgr2=0x7f2268073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:06.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.037+0000 7f226d053640 1 -- 172.21.15.16:0/1236239176 shutdown_connections 2024-04-24T23:49:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.037+0000 7f226d053640 1 -- 172.21.15.16:0/1236239176 wait complete. 2024-04-24T23:49:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.039+0000 7f226d053640 1 Processor -- start 2024-04-24T23:49:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.039+0000 7f226d053640 1 -- start start 2024-04-24T23:49:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.039+0000 7f226d053640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2268151960 0x7f22680b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.039+0000 7f226d053640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2268156740 0x7f22680ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.039+0000 7f226d053640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2268161980 0x7f2268161e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.039+0000 7f226d053640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22680d29c0 con 0x7f2268156740 2024-04-24T23:49:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.039+0000 7f226d053640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f22680d2840 con 0x7f2268151960 2024-04-24T23:49:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.039+0000 7f226d053640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f22680d16b0 con 0x7f2268161980 2024-04-24T23:49:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f225ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2268156740 0x7f22680ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f225ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2268156740 0x7f22680ca7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:43618/0 (socket says 172.21.15.16:43618) 2024-04-24T23:49:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f225ffff640 1 -- 172.21.15.16:0/768735707 learned_addr learned my addr 172.21.15.16:0/768735707 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f226759e640 1 --2- 172.21.15.16:0/768735707 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2268161980 0x7f2268161e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:06.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f2266d9d640 1 --2- 172.21.15.16:0/768735707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2268151960 0x7f22680b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f225ffff640 1 -- 172.21.15.16:0/768735707 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2268161980 msgr2=0x7f2268161e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f225ffff640 1 --2- 172.21.15.16:0/768735707 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2268161980 0x7f2268161e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f225ffff640 1 -- 172.21.15.16:0/768735707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2268151960 msgr2=0x7f22680b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f225ffff640 1 --2- 172.21.15.16:0/768735707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2268151960 0x7f22680b3da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f225ffff640 1 -- 172.21.15.16:0/768735707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22680b4d70 con 0x7f2268156740 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f225ffff640 1 --2- 172.21.15.16:0/768735707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2268156740 0x7f22680ca7e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2250011970 tx=0x7f2250011e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f225f7fe640 1 -- 172.21.15.16:0/768735707 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f225000faf0 con 0x7f2268156740 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f225f7fe640 1 -- 172.21.15.16:0/768735707 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f225000fc90 con 0x7f2268156740 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f226d053640 1 -- 172.21.15.16:0/768735707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22680b44d0 con 0x7f2268156740 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f225f7fe640 1 -- 172.21.15.16:0/768735707 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2250014440 con 0x7f2268156740 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.040+0000 7f226d053640 1 -- 172.21.15.16:0/768735707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22680b4a90 con 0x7f2268156740 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.041+0000 7f226d053640 1 -- 172.21.15.16:0/768735707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2268068f90 con 0x7f2268156740 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.042+0000 7f225f7fe640 1 -- 172.21.15.16:0/768735707 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2250012040 con 0x7f2268156740 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.042+0000 7f225f7fe640 1 --2- 172.21.15.16:0/768735707 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f224806be30 0x7f224806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.042+0000 7f225f7fe640 1 -- 172.21.15.16:0/768735707 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f225008b670 con 0x7f2268156740 2024-04-24T23:49:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.042+0000 7f2266d9d640 1 --2- 172.21.15.16:0/768735707 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f224806be30 0x7f224806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.043+0000 7f2266d9d640 1 --2- 172.21.15.16:0/768735707 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f224806be30 0x7f224806e2f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2258000ff0 tx=0x7f2258001e50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:06.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.046+0000 7f225f7fe640 1 -- 172.21.15.16:0/768735707 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2250019030 con 0x7f2268156740 2024-04-24T23:49:06.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.230+0000 7f226d053640 1 -- 172.21.15.16:0/768735707 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f22680b3b80 con 0x7f2268156740 2024-04-24T23:49:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.231+0000 7f225f7fe640 1 -- 172.21.15.16:0/768735707 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f2250059720 con 0x7f2268156740 2024-04-24T23:49:06.859 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.233+0000 7f226d053640 1 -- 172.21.15.16:0/768735707 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f224806be30 msgr2=0x7f224806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.233+0000 7f226d053640 1 --2- 172.21.15.16:0/768735707 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f224806be30 0x7f224806e2f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2258000ff0 tx=0x7f2258001e50 comp rx=0 tx=0).stop 2024-04-24T23:49:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.233+0000 7f226d053640 1 -- 172.21.15.16:0/768735707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2268156740 msgr2=0x7f22680ca7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.233+0000 7f226d053640 1 --2- 172.21.15.16:0/768735707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2268156740 0x7f22680ca7e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2250011970 tx=0x7f2250011e30 comp rx=0 tx=0).stop 2024-04-24T23:49:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.233+0000 7f226d053640 1 -- 172.21.15.16:0/768735707 shutdown_connections 2024-04-24T23:49:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.233+0000 7f226d053640 1 --2- 172.21.15.16:0/768735707 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f224806be30 0x7f224806e2f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.233+0000 7f226d053640 1 --2- 172.21.15.16:0/768735707 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2268161980 0x7f2268161e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.233+0000 7f226d053640 1 --2- 172.21.15.16:0/768735707 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2268156740 0x7f22680ca7e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.233+0000 7f226d053640 1 --2- 172.21.15.16:0/768735707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2268151960 0x7f22680b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.233+0000 7f226d053640 1 -- 172.21.15.16:0/768735707 >> 172.21.15.16:0/768735707 conn(0x7f2268072e70 msgr2=0x7f22680b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.233+0000 7f226d053640 1 -- 172.21.15.16:0/768735707 shutdown_connections 2024-04-24T23:49:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.233+0000 7f226d053640 1 -- 172.21.15.16:0/768735707 wait complete. 2024-04-24T23:49:06.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.975+0000 7fc34d064640 1 Processor -- start 2024-04-24T23:49:06.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.975+0000 7fc34d064640 1 -- start start 2024-04-24T23:49:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.976+0000 7fc34d064640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc34806bc40 0x7fc34806c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.976+0000 7fc34d064640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3480b7050 0x7fc3480b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.976+0000 7fc34d064640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3480d24d0 con 0x7fc34815dfd0 2024-04-24T23:49:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.976+0000 7fc34d064640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fc34806ba00 con 0x7fc34806bc40 2024-04-24T23:49:06.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.976+0000 7fc34d064640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fc3480d27b0 con 0x7fc3480b7050 2024-04-24T23:49:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.976+0000 7fc34759e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3480b7050 0x7fc3480b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.976+0000 7fc34659c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc34806bc40 0x7fc34806c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.976+0000 7fc346d9d640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fc34815dfd0 0x7fc34806b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:52136/0 (socket says 172.21.15.16:52136) 2024-04-24T23:49:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.976+0000 7fc346d9d640 1 -- 172.21.15.16:0/2052190355 learned_addr learned my addr 172.21.15.16:0/2052190355 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.977+0000 7fc34759e640 1 -- 172.21.15.16:0/2052190355 >> v2:172.21.15.156:3300/0 conn(0x7fc34806bc40 msgr2=0x7fc34806c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.977+0000 7fc34759e640 1 --2- 172.21.15.16:0/2052190355 >> v2:172.21.15.156:3300/0 conn(0x7fc34806bc40 0x7fc34806c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.977+0000 7fc34759e640 1 -- 172.21.15.16:0/2052190355 >> v1:172.21.15.156:6789/0 conn(0x7fc34815dfd0 legacy=0x7fc34806b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.977+0000 7fc345d9b640 1 -- 172.21.15.16:0/2052190355 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775411485 0 0) 0x7fc3480d24d0 con 0x7fc34815dfd0 2024-04-24T23:49:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.977+0000 7fc34759e640 1 -- 172.21.15.16:0/2052190355 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3480d2a90 con 0x7fc3480b7050 2024-04-24T23:49:06.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.977+0000 7fc34659c640 1 --2- 172.21.15.16:0/2052190355 >> v2:172.21.15.156:3300/0 conn(0x7fc34806bc40 0x7fc34806c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.978+0000 7fc34759e640 1 --2- 172.21.15.16:0/2052190355 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3480b7050 0x7fc3480b74a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc338009d90 tx=0x7fc33802a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51b37852747459c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.978+0000 7fc345d9b640 1 -- 172.21.15.16:0/2052190355 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc338031600 con 0x7fc3480b7050 2024-04-24T23:49:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.978+0000 7fc345d9b640 1 -- 172.21.15.16:0/2052190355 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3380317a0 con 0x7fc3480b7050 2024-04-24T23:49:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.978+0000 7fc345d9b640 1 -- 172.21.15.16:0/2052190355 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc338031b70 con 0x7fc3480b7050 2024-04-24T23:49:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.978+0000 7fc34d064640 1 -- 172.21.15.16:0/2052190355 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3480b7050 msgr2=0x7fc3480b74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.978+0000 7fc34d064640 1 --2- 172.21.15.16:0/2052190355 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3480b7050 0x7fc3480b74a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc338009d90 tx=0x7fc33802a5d0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.978+0000 7fc34d064640 1 -- 172.21.15.16:0/2052190355 shutdown_connections 2024-04-24T23:49:06.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.978+0000 7fc34d064640 1 --2- 172.21.15.16:0/2052190355 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3480b7050 0x7fc3480b74a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.978+0000 7fc34d064640 1 --2- 172.21.15.16:0/2052190355 >> v2:172.21.15.156:3300/0 conn(0x7fc34806bc40 0x7fc34806c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.978+0000 7fc34d064640 1 -- 172.21.15.16:0/2052190355 >> 172.21.15.16:0/2052190355 conn(0x7fc3480d3dc0 msgr2=0x7fc3480d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:06.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.978+0000 7fc34d064640 1 -- 172.21.15.16:0/2052190355 shutdown_connections 2024-04-24T23:49:06.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.978+0000 7fc34d064640 1 -- 172.21.15.16:0/2052190355 wait complete. 2024-04-24T23:49:06.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.980+0000 7fc34d064640 1 Processor -- start 2024-04-24T23:49:06.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.980+0000 7fc34d064640 1 -- start start 2024-04-24T23:49:06.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc34d064640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc34806bc40 0x7fc34814b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:06.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc34d064640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3480b7050 0x7fc348149e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:06.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc34d064640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc34815dfd0 0x7fc34814beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:06.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc34d064640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc3480cead0 con 0x7fc34806bc40 2024-04-24T23:49:06.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc34d064640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc3480d3050 con 0x7fc3480b7050 2024-04-24T23:49:06.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc34d064640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fc3480b79e0 con 0x7fc34815dfd0 2024-04-24T23:49:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc346d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc34806bc40 0x7fc34814b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc346d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc34806bc40 0x7fc34814b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44216/0 (socket says 172.21.15.16:44216) 2024-04-24T23:49:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc346d9d640 1 -- 172.21.15.16:0/2836251944 learned_addr learned my addr 172.21.15.16:0/2836251944 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc34759e640 1 --2- 172.21.15.16:0/2836251944 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc34815dfd0 0x7fc34814beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc34659c640 1 --2- 172.21.15.16:0/2836251944 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3480b7050 0x7fc348149e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc346d9d640 1 -- 172.21.15.16:0/2836251944 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc34815dfd0 msgr2=0x7fc34814beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc346d9d640 1 --2- 172.21.15.16:0/2836251944 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc34815dfd0 0x7fc34814beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc346d9d640 1 -- 172.21.15.16:0/2836251944 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3480b7050 msgr2=0x7fc348149e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc346d9d640 1 --2- 172.21.15.16:0/2836251944 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3480b7050 0x7fc348149e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc346d9d640 1 -- 172.21.15.16:0/2836251944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc34814a6c0 con 0x7fc34806bc40 2024-04-24T23:49:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.981+0000 7fc346d9d640 1 --2- 172.21.15.16:0/2836251944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc34806bc40 0x7fc34814b760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc34000c430 tx=0x7fc34000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.982+0000 7fc33f7fe640 1 -- 172.21.15.16:0/2836251944 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3400074e0 con 0x7fc34806bc40 2024-04-24T23:49:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.982+0000 7fc33f7fe640 1 -- 172.21.15.16:0/2836251944 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3400047e0 con 0x7fc34806bc40 2024-04-24T23:49:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.982+0000 7fc34d064640 1 -- 172.21.15.16:0/2836251944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc34814a9a0 con 0x7fc34806bc40 2024-04-24T23:49:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.982+0000 7fc33f7fe640 1 -- 172.21.15.16:0/2836251944 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc340004bd0 con 0x7fc34806bc40 2024-04-24T23:49:06.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.982+0000 7fc34d064640 1 -- 172.21.15.16:0/2836251944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc34814af60 con 0x7fc34806bc40 2024-04-24T23:49:06.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.983+0000 7fc34d064640 1 -- 172.21.15.16:0/2836251944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc348068f90 con 0x7fc34806bc40 2024-04-24T23:49:06.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.983+0000 7fc33f7fe640 1 -- 172.21.15.16:0/2836251944 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc340011070 con 0x7fc34806bc40 2024-04-24T23:49:06.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.984+0000 7fc33f7fe640 1 --2- 172.21.15.16:0/2836251944 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc32806be30 0x7fc32806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:06.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.984+0000 7fc33f7fe640 1 -- 172.21.15.16:0/2836251944 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc34008ac50 con 0x7fc34806bc40 2024-04-24T23:49:06.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.984+0000 7fc34659c640 1 --2- 172.21.15.16:0/2836251944 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc32806be30 0x7fc32806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:06.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.984+0000 7fc34659c640 1 --2- 172.21.15.16:0/2836251944 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc32806be30 0x7fc32806e2f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc330009560 tx=0x7fc330009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:06.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:06.988+0000 7fc33f7fe640 1 -- 172.21.15.16:0/2836251944 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc340090020 con 0x7fc34806bc40 2024-04-24T23:49:07.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.169+0000 7fc34d064640 1 -- 172.21.15.16:0/2836251944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc348160ce0 con 0x7fc34806bc40 2024-04-24T23:49:07.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.170+0000 7fc33f7fe640 1 -- 172.21.15.16:0/2836251944 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7fc340058d00 con 0x7fc34806bc40 2024-04-24T23:49:07.170 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:07.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.172+0000 7fc34d064640 1 -- 172.21.15.16:0/2836251944 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc32806be30 msgr2=0x7fc32806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.172+0000 7fc34d064640 1 --2- 172.21.15.16:0/2836251944 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc32806be30 0x7fc32806e2f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc330009560 tx=0x7fc330009190 comp rx=0 tx=0).stop 2024-04-24T23:49:07.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.172+0000 7fc34d064640 1 -- 172.21.15.16:0/2836251944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc34806bc40 msgr2=0x7fc34814b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.172+0000 7fc34d064640 1 --2- 172.21.15.16:0/2836251944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc34806bc40 0x7fc34814b760 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc34000c430 tx=0x7fc34000c8f0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.172+0000 7fc34d064640 1 -- 172.21.15.16:0/2836251944 shutdown_connections 2024-04-24T23:49:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.172+0000 7fc34d064640 1 --2- 172.21.15.16:0/2836251944 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc32806be30 0x7fc32806e2f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.172+0000 7fc34d064640 1 --2- 172.21.15.16:0/2836251944 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc34815dfd0 0x7fc34814beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.172+0000 7fc34d064640 1 --2- 172.21.15.16:0/2836251944 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc3480b7050 0x7fc348149e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.172+0000 7fc34d064640 1 --2- 172.21.15.16:0/2836251944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc34806bc40 0x7fc34814b760 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.172+0000 7fc34d064640 1 -- 172.21.15.16:0/2836251944 >> 172.21.15.16:0/2836251944 conn(0x7fc3480d3dc0 msgr2=0x7fc3480b58a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.172+0000 7fc34d064640 1 -- 172.21.15.16:0/2836251944 shutdown_connections 2024-04-24T23:49:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.172+0000 7fc34d064640 1 -- 172.21.15.16:0/2836251944 wait complete. 2024-04-24T23:49:07.184 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:49:07.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.287+0000 7f8b2c329640 1 Processor -- start 2024-04-24T23:49:07.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.287+0000 7f8b2c329640 1 -- start start 2024-04-24T23:49:07.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2c329640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8b2415a450 0x7f8b240b2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2c329640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8b2406e910 0x7f8b24072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2c329640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b240727d0 0x7f8b2415deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2c329640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8b24072c20 con 0x7f8b240727d0 2024-04-24T23:49:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2c329640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8b240b3300 con 0x7f8b2406e910 2024-04-24T23:49:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2c329640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8b240b35e0 con 0x7f8b2415a450 2024-04-24T23:49:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2a8c6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b240727d0 0x7f8b2415deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2a8c6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b240727d0 0x7f8b2415deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44218/0 (socket says 172.21.15.16:44218) 2024-04-24T23:49:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2a8c6640 1 -- 172.21.15.16:0/2189161245 learned_addr learned my addr 172.21.15.16:0/2189161245 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b298c4640 1 --2- 172.21.15.16:0/2189161245 >> v2:172.21.15.156:3300/0 conn(0x7f8b2406e910 0x7f8b24072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2a0c5640 1 --2- 172.21.15.16:0/2189161245 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8b2415a450 0x7f8b240b2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2a8c6640 1 -- 172.21.15.16:0/2189161245 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8b2415a450 msgr2=0x7f8b240b2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2a8c6640 1 --2- 172.21.15.16:0/2189161245 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8b2415a450 0x7f8b240b2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2a8c6640 1 -- 172.21.15.16:0/2189161245 >> v2:172.21.15.156:3300/0 conn(0x7f8b2406e910 msgr2=0x7f8b24072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:07.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2a8c6640 1 --2- 172.21.15.16:0/2189161245 >> v2:172.21.15.156:3300/0 conn(0x7f8b2406e910 0x7f8b24072290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.288+0000 7f8b2a8c6640 1 -- 172.21.15.16:0/2189161245 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b240cdd10 con 0x7f8b240727d0 2024-04-24T23:49:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.289+0000 7f8b2a8c6640 1 --2- 172.21.15.16:0/2189161245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b240727d0 0x7f8b2415deb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8b240d0460 tx=0x7f8b1402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be15a32f04c508c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.289+0000 7f8b290c3640 1 -- 172.21.15.16:0/2189161245 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b14037070 con 0x7f8b240727d0 2024-04-24T23:49:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.289+0000 7f8b290c3640 1 -- 172.21.15.16:0/2189161245 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b14032440 con 0x7f8b240727d0 2024-04-24T23:49:07.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.289+0000 7f8b290c3640 1 -- 172.21.15.16:0/2189161245 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b14032810 con 0x7f8b240727d0 2024-04-24T23:49:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.289+0000 7f8b2c329640 1 -- 172.21.15.16:0/2189161245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b240727d0 msgr2=0x7f8b2415deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.289+0000 7f8b2c329640 1 --2- 172.21.15.16:0/2189161245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b240727d0 0x7f8b2415deb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8b240d0460 tx=0x7f8b1402a140 comp rx=0 tx=0).stop 2024-04-24T23:49:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.290+0000 7f8b2c329640 1 -- 172.21.15.16:0/2189161245 shutdown_connections 2024-04-24T23:49:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.290+0000 7f8b2c329640 1 --2- 172.21.15.16:0/2189161245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b240727d0 0x7f8b2415deb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.290+0000 7f8b2c329640 1 --2- 172.21.15.16:0/2189161245 >> v2:172.21.15.156:3300/0 conn(0x7f8b2406e910 0x7f8b24072290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.290+0000 7f8b2c329640 1 --2- 172.21.15.16:0/2189161245 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8b2415a450 0x7f8b240b2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.290+0000 7f8b2c329640 1 -- 172.21.15.16:0/2189161245 >> 172.21.15.16:0/2189161245 conn(0x7f8b240b2000 msgr2=0x7f8b240cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.290+0000 7f8b2c329640 1 -- 172.21.15.16:0/2189161245 shutdown_connections 2024-04-24T23:49:07.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.290+0000 7f8b2c329640 1 -- 172.21.15.16:0/2189161245 wait complete. 2024-04-24T23:49:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.291+0000 7f8b2c329640 1 Processor -- start 2024-04-24T23:49:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.291+0000 7f8b2c329640 1 -- start start 2024-04-24T23:49:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.291+0000 7f8b2c329640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8b2406e910 0x7f8b241510a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.291+0000 7f8b2c329640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b240727d0 0x7f8b240ae2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.291+0000 7f8b2c329640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b2415a450 0x7f8b241e71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.291+0000 7f8b2c329640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2406bb60 con 0x7f8b240727d0 2024-04-24T23:49:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.291+0000 7f8b2c329640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8b240cc400 con 0x7f8b2415a450 2024-04-24T23:49:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.291+0000 7f8b2c329640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8b240b2f60 con 0x7f8b2406e910 2024-04-24T23:49:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b298c4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b240727d0 0x7f8b240ae2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b298c4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b240727d0 0x7f8b240ae2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44222/0 (socket says 172.21.15.16:44222) 2024-04-24T23:49:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b298c4640 1 -- 172.21.15.16:0/1188517384 learned_addr learned my addr 172.21.15.16:0/1188517384 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b2a0c5640 1 --2- 172.21.15.16:0/1188517384 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8b2406e910 0x7f8b241510a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b2a8c6640 1 --2- 172.21.15.16:0/1188517384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b2415a450 0x7f8b241e71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b298c4640 1 -- 172.21.15.16:0/1188517384 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8b2406e910 msgr2=0x7f8b241510a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b298c4640 1 --2- 172.21.15.16:0/1188517384 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8b2406e910 0x7f8b241510a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b298c4640 1 -- 172.21.15.16:0/1188517384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b2415a450 msgr2=0x7f8b241e71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b298c4640 1 --2- 172.21.15.16:0/1188517384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b2415a450 0x7f8b241e71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b298c4640 1 -- 172.21.15.16:0/1188517384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b240aeb40 con 0x7f8b240727d0 2024-04-24T23:49:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b298c4640 1 --2- 172.21.15.16:0/1188517384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b240727d0 0x7f8b240ae2a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800e840 tx=0x7f8b1800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b22ffd640 1 -- 172.21.15.16:0/1188517384 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18017070 con 0x7f8b240727d0 2024-04-24T23:49:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b22ffd640 1 -- 172.21.15.16:0/1188517384 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b1800c730 con 0x7f8b240727d0 2024-04-24T23:49:07.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b2c329640 1 -- 172.21.15.16:0/1188517384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b240aee20 con 0x7f8b240727d0 2024-04-24T23:49:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b22ffd640 1 -- 172.21.15.16:0/1188517384 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1800cb20 con 0x7f8b240727d0 2024-04-24T23:49:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.292+0000 7f8b2c329640 1 -- 172.21.15.16:0/1188517384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b24154c60 con 0x7f8b240727d0 2024-04-24T23:49:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.293+0000 7f8b2c329640 1 -- 172.21.15.16:0/1188517384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b240731f0 con 0x7f8b240727d0 2024-04-24T23:49:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.294+0000 7f8b22ffd640 1 -- 172.21.15.16:0/1188517384 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8b1801a070 con 0x7f8b240727d0 2024-04-24T23:49:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.294+0000 7f8b22ffd640 1 --2- 172.21.15.16:0/1188517384 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8b1006be30 0x7f8b1006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.294+0000 7f8b22ffd640 1 -- 172.21.15.16:0/1188517384 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8b1808b210 con 0x7f8b240727d0 2024-04-24T23:49:07.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.294+0000 7f8b2a0c5640 1 --2- 172.21.15.16:0/1188517384 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8b1006be30 0x7f8b1006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.295+0000 7f8b2a0c5640 1 --2- 172.21.15.16:0/1188517384 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8b1006be30 0x7f8b1006e2f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c0113e0 tx=0x7f8b1c00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:07.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.298+0000 7f8b22ffd640 1 -- 172.21.15.16:0/1188517384 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8b180592c0 con 0x7f8b240727d0 2024-04-24T23:49:07.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.494+0000 7f8b2c329640 1 -- 172.21.15.16:0/1188517384 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8b241e6fa0 con 0x7f8b240727d0 2024-04-24T23:49:07.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.495+0000 7f8b22ffd640 1 -- 172.21.15.16:0/1188517384 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7f8b1805d5c0 con 0x7f8b240727d0 2024-04-24T23:49:07.495 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-04-24T23:49:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.497+0000 7f8b2c329640 1 -- 172.21.15.16:0/1188517384 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8b1006be30 msgr2=0x7f8b1006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.497+0000 7f8b2c329640 1 --2- 172.21.15.16:0/1188517384 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8b1006be30 0x7f8b1006e2f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c0113e0 tx=0x7f8b1c00b040 comp rx=0 tx=0).stop 2024-04-24T23:49:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.497+0000 7f8b2c329640 1 -- 172.21.15.16:0/1188517384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b240727d0 msgr2=0x7f8b240ae2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.497+0000 7f8b2c329640 1 --2- 172.21.15.16:0/1188517384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b240727d0 0x7f8b240ae2a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800e840 tx=0x7f8b1800ed00 comp rx=0 tx=0).stop 2024-04-24T23:49:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.497+0000 7f8b2c329640 1 -- 172.21.15.16:0/1188517384 shutdown_connections 2024-04-24T23:49:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.497+0000 7f8b2c329640 1 --2- 172.21.15.16:0/1188517384 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8b1006be30 0x7f8b1006e2f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.497+0000 7f8b2c329640 1 --2- 172.21.15.16:0/1188517384 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8b2415a450 0x7f8b241e71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.497+0000 7f8b2c329640 1 --2- 172.21.15.16:0/1188517384 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b240727d0 0x7f8b240ae2a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.497+0000 7f8b2c329640 1 --2- 172.21.15.16:0/1188517384 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8b2406e910 0x7f8b241510a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.497+0000 7f8b2c329640 1 -- 172.21.15.16:0/1188517384 >> 172.21.15.16:0/1188517384 conn(0x7f8b240b2000 msgr2=0x7f8b240af690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.497+0000 7f8b2c329640 1 -- 172.21.15.16:0/1188517384 shutdown_connections 2024-04-24T23:49:07.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.497+0000 7f8b2c329640 1 -- 172.21.15.16:0/1188517384 wait complete. 2024-04-24T23:49:07.507 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:49:07.507 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:49:07.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.665+0000 7f7ab9363640 1 Processor -- start 2024-04-24T23:49:07.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.665+0000 7f7ab9363640 1 -- start start 2024-04-24T23:49:07.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab9363640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab415ed10 0x7f7ab415e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab9363640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ab40d0370 con 0x7f7ab415e930 2024-04-24T23:49:07.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab9363640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ab406bcd0 con 0x7f7ab40cff10 2024-04-24T23:49:07.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab9363640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7ab406bfb0 con 0x7f7ab415ed10 2024-04-24T23:49:07.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab2ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7ab415e930 0x7f7ab415dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49666/0 (socket says 172.21.15.16:49666) 2024-04-24T23:49:07.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab2ffd640 1 -- 172.21.15.16:0/1937974017 learned_addr learned my addr 172.21.15.16:0/1937974017 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:07.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab27fc640 1 --2- 172.21.15.16:0/1937974017 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab415ed10 0x7f7ab415e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab1ffb640 1 -- 172.21.15.16:0/1937974017 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 718269107 0 0) 0x7f7aa4001110 con 0x7f7ab415e930 2024-04-24T23:49:07.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab1ffb640 1 -- 172.21.15.16:0/1937974017 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a9c003520 con 0x7f7ab415e930 2024-04-24T23:49:07.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab1ffb640 1 -- 172.21.15.16:0/1937974017 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 276421338 0 0) 0x7f7ab406bcd0 con 0x7f7ab40cff10 2024-04-24T23:49:07.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab1ffb640 1 -- 172.21.15.16:0/1937974017 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7aa4001110 con 0x7f7ab40cff10 2024-04-24T23:49:07.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab1ffb640 1 -- 172.21.15.16:0/1937974017 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2918479903 0 0) 0x7f7a9c003520 con 0x7f7ab415e930 2024-04-24T23:49:07.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab27fc640 1 -- 172.21.15.16:0/1937974017 >> v1:172.21.15.156:6789/0 conn(0x7f7ab40cff10 legacy=0x7f7ab40d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab27fc640 1 -- 172.21.15.16:0/1937974017 >> v1:172.21.15.16:6789/0 conn(0x7f7ab415e930 legacy=0x7f7ab415dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.666+0000 7f7ab27fc640 1 -- 172.21.15.16:0/1937974017 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab406c290 con 0x7f7ab415ed10 2024-04-24T23:49:07.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.667+0000 7f7ab27fc640 1 --2- 172.21.15.16:0/1937974017 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab415ed10 0x7f7ab415e1d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7aa80097d0 tx=0x7f7aa802a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c129021551eeafb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:07.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.667+0000 7f7ab1ffb640 1 -- 172.21.15.16:0/1937974017 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa8037070 con 0x7f7ab415ed10 2024-04-24T23:49:07.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.667+0000 7f7ab1ffb640 1 -- 172.21.15.16:0/1937974017 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aa802ac30 con 0x7f7ab415ed10 2024-04-24T23:49:07.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.667+0000 7f7ab1ffb640 1 -- 172.21.15.16:0/1937974017 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4001840 con 0x7f7ab415ed10 2024-04-24T23:49:07.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.668+0000 7f7ab9363640 1 -- 172.21.15.16:0/1937974017 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab415ed10 msgr2=0x7f7ab415e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.668+0000 7f7ab9363640 1 --2- 172.21.15.16:0/1937974017 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab415ed10 0x7f7ab415e1d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7aa80097d0 tx=0x7f7aa802a1a0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.668+0000 7f7ab9363640 1 -- 172.21.15.16:0/1937974017 shutdown_connections 2024-04-24T23:49:07.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.668+0000 7f7ab9363640 1 --2- 172.21.15.16:0/1937974017 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab415ed10 0x7f7ab415e1d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.668+0000 7f7ab9363640 1 -- 172.21.15.16:0/1937974017 >> 172.21.15.16:0/1937974017 conn(0x7f7ab40d4050 msgr2=0x7f7ab40d4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:07.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.668+0000 7f7ab9363640 1 -- 172.21.15.16:0/1937974017 shutdown_connections 2024-04-24T23:49:07.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.668+0000 7f7ab9363640 1 -- 172.21.15.16:0/1937974017 wait complete. 2024-04-24T23:49:07.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.669+0000 7f7ab9363640 1 Processor -- start 2024-04-24T23:49:07.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.669+0000 7f7ab9363640 1 -- start start 2024-04-24T23:49:07.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.670+0000 7f7ab9363640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab40cff10 0x7f7ab41f3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.670+0000 7f7ab9363640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab415e930 0x7f7ab4161e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.670+0000 7f7ab9363640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ab415ed10 0x7f7ab415ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.670+0000 7f7ab9363640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab406b5a0 con 0x7f7ab40cff10 2024-04-24T23:49:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.670+0000 7f7ab9363640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7ab406c850 con 0x7f7ab415ed10 2024-04-24T23:49:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.670+0000 7f7ab9363640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7ab406b720 con 0x7f7ab415e930 2024-04-24T23:49:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.670+0000 7f7ab2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab40cff10 0x7f7ab41f3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.670+0000 7f7ab2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab40cff10 0x7f7ab41f3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44234/0 (socket says 172.21.15.16:44234) 2024-04-24T23:49:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.670+0000 7f7ab2ffd640 1 -- 172.21.15.16:0/401167059 learned_addr learned my addr 172.21.15.16:0/401167059 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.670+0000 7f7ab27fc640 1 --2- 172.21.15.16:0/401167059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab415e930 0x7f7ab4161e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.670+0000 7f7ab37fe640 1 --2- 172.21.15.16:0/401167059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ab415ed10 0x7f7ab415ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.670+0000 7f7ab2ffd640 1 -- 172.21.15.16:0/401167059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab415e930 msgr2=0x7f7ab4161e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.670+0000 7f7ab2ffd640 1 --2- 172.21.15.16:0/401167059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab415e930 0x7f7ab4161e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.671+0000 7f7ab2ffd640 1 -- 172.21.15.16:0/401167059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ab415ed10 msgr2=0x7f7ab415ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.671+0000 7f7ab2ffd640 1 --2- 172.21.15.16:0/401167059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ab415ed10 0x7f7ab415ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.671+0000 7f7ab2ffd640 1 -- 172.21.15.16:0/401167059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ab41623d0 con 0x7f7ab40cff10 2024-04-24T23:49:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.671+0000 7f7ab2ffd640 1 --2- 172.21.15.16:0/401167059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab40cff10 0x7f7ab41f3dd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4000fd0 tx=0x7f7aa400cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.671+0000 7f7a9b7fe640 1 -- 172.21.15.16:0/401167059 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa4017070 con 0x7f7ab40cff10 2024-04-24T23:49:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.671+0000 7f7a9b7fe640 1 -- 172.21.15.16:0/401167059 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7aa4005d00 con 0x7f7ab40cff10 2024-04-24T23:49:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.671+0000 7f7a9b7fe640 1 -- 172.21.15.16:0/401167059 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aa40072b0 con 0x7f7ab40cff10 2024-04-24T23:49:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.671+0000 7f7ab9363640 1 -- 172.21.15.16:0/401167059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ab41f44d0 con 0x7f7ab40cff10 2024-04-24T23:49:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.671+0000 7f7ab9363640 1 -- 172.21.15.16:0/401167059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ab41f4a90 con 0x7f7ab40cff10 2024-04-24T23:49:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.672+0000 7f7ab9363640 1 -- 172.21.15.16:0/401167059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ab406dfe0 con 0x7f7ab40cff10 2024-04-24T23:49:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.672+0000 7f7a9b7fe640 1 -- 172.21.15.16:0/401167059 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7aa401a070 con 0x7f7ab40cff10 2024-04-24T23:49:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.673+0000 7f7a9b7fe640 1 --2- 172.21.15.16:0/401167059 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7aa006be30 0x7f7aa006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.673+0000 7f7a9b7fe640 1 -- 172.21.15.16:0/401167059 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7aa408a7f0 con 0x7f7ab40cff10 2024-04-24T23:49:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.673+0000 7f7ab27fc640 1 --2- 172.21.15.16:0/401167059 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7aa006be30 0x7f7aa006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.674+0000 7f7ab27fc640 1 --2- 172.21.15.16:0/401167059 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7aa006be30 0x7f7aa006e2f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8034000 tx=0x7f7aa802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.677+0000 7f7a9b7fe640 1 -- 172.21.15.16:0/401167059 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7ab40d0370 con 0x7f7ab40cff10 2024-04-24T23:49:07.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.864+0000 7f7ab9363640 1 -- 172.21.15.16:0/401167059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7ab404f5e0 con 0x7f7ab40cff10 2024-04-24T23:49:07.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.865+0000 7f7a9b7fe640 1 -- 172.21.15.16:0/401167059 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18502 (secure 0 0 0) 0x7f7aa4058ef0 con 0x7f7ab40cff10 2024-04-24T23:49:07.865 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:07.865 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:59.888015+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.867+0000 7f7ab9363640 1 -- 172.21.15.16:0/401167059 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7aa006be30 msgr2=0x7f7aa006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.867+0000 7f7ab9363640 1 --2- 172.21.15.16:0/401167059 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7aa006be30 0x7f7aa006e2f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8034000 tx=0x7f7aa802e000 comp rx=0 tx=0).stop 2024-04-24T23:49:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.867+0000 7f7ab9363640 1 -- 172.21.15.16:0/401167059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab40cff10 msgr2=0x7f7ab41f3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.867+0000 7f7ab9363640 1 --2- 172.21.15.16:0/401167059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab40cff10 0x7f7ab41f3dd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7aa4000fd0 tx=0x7f7aa400cf50 comp rx=0 tx=0).stop 2024-04-24T23:49:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.867+0000 7f7ab9363640 1 -- 172.21.15.16:0/401167059 shutdown_connections 2024-04-24T23:49:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.867+0000 7f7ab9363640 1 --2- 172.21.15.16:0/401167059 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7aa006be30 0x7f7aa006e2f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.867+0000 7f7ab9363640 1 --2- 172.21.15.16:0/401167059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ab415ed10 0x7f7ab415ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.867+0000 7f7ab9363640 1 --2- 172.21.15.16:0/401167059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7ab415e930 0x7f7ab4161e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.867+0000 7f7ab9363640 1 --2- 172.21.15.16:0/401167059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ab40cff10 0x7f7ab41f3dd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.867+0000 7f7ab9363640 1 -- 172.21.15.16:0/401167059 >> 172.21.15.16:0/401167059 conn(0x7f7ab40d4050 msgr2=0x7f7ab40d4430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:07.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.867+0000 7f7ab9363640 1 -- 172.21.15.16:0/401167059 shutdown_connections 2024-04-24T23:49:07.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.867+0000 7f7ab9363640 1 -- 172.21.15.16:0/401167059 wait complete. 2024-04-24T23:49:07.878 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:07.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:07.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.983+0000 7f604bb70640 1 Processor -- start 2024-04-24T23:49:07.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.983+0000 7f604bb70640 1 -- start start 2024-04-24T23:49:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.984+0000 7f604bb70640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f604415d0a0 0x7f60440ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.984+0000 7f604bb70640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60440ceca0 0x7f604406ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.984+0000 7f604bb70640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f604415f9c0 con 0x7f60441455f0 2024-04-24T23:49:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.984+0000 7f604bb70640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f60440cf100 con 0x7f60440ceca0 2024-04-24T23:49:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.984+0000 7f604bb70640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f60441605d0 con 0x7f604415d0a0 2024-04-24T23:49:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.984+0000 7f604a10d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60440ceca0 0x7f604406ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.984+0000 7f604a10d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60440ceca0 0x7f604406ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44242/0 (socket says 172.21.15.16:44242) 2024-04-24T23:49:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.984+0000 7f604a10d640 1 -- 172.21.15.16:0/3027277471 learned_addr learned my addr 172.21.15.16:0/3027277471 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.984+0000 7f604910b640 1 --2- 172.21.15.16:0/3027277471 >> v2:172.21.15.156:3300/0 conn(0x7f604415d0a0 0x7f60440ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.984+0000 7f604a10d640 1 -- 172.21.15.16:0/3027277471 >> v2:172.21.15.156:3300/0 conn(0x7f604415d0a0 msgr2=0x7f60440ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.984+0000 7f604a10d640 1 --2- 172.21.15.16:0/3027277471 >> v2:172.21.15.156:3300/0 conn(0x7f604415d0a0 0x7f60440ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.984+0000 7f604a10d640 1 -- 172.21.15.16:0/3027277471 >> v1:172.21.15.16:6789/0 conn(0x7f60441455f0 legacy=0x7f604406b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.984+0000 7f604a10d640 1 -- 172.21.15.16:0/3027277471 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604415fca0 con 0x7f60440ceca0 2024-04-24T23:49:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.984+0000 7f604890a640 1 -- 172.21.15.16:0/3027277471 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1179137620 0 0) 0x7f604415f9c0 con 0x7f60441455f0 2024-04-24T23:49:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.985+0000 7f604a10d640 1 --2- 172.21.15.16:0/3027277471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60440ceca0 0x7f604406ccc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f60440cf340 tx=0x7f603c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8508402dad62a0f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.985+0000 7f604890a640 1 -- 172.21.15.16:0/3027277471 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c036070 con 0x7f60440ceca0 2024-04-24T23:49:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.985+0000 7f604890a640 1 -- 172.21.15.16:0/3027277471 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f603c031600 con 0x7f60440ceca0 2024-04-24T23:49:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.985+0000 7f604890a640 1 -- 172.21.15.16:0/3027277471 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c0319d0 con 0x7f60440ceca0 2024-04-24T23:49:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.986+0000 7f604bb70640 1 -- 172.21.15.16:0/3027277471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60440ceca0 msgr2=0x7f604406ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.986+0000 7f604bb70640 1 --2- 172.21.15.16:0/3027277471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60440ceca0 0x7f604406ccc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f60440cf340 tx=0x7f603c02a6a0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.986+0000 7f604bb70640 1 -- 172.21.15.16:0/3027277471 shutdown_connections 2024-04-24T23:49:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.986+0000 7f604bb70640 1 --2- 172.21.15.16:0/3027277471 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60440ceca0 0x7f604406ccc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.986+0000 7f604bb70640 1 --2- 172.21.15.16:0/3027277471 >> v2:172.21.15.156:3300/0 conn(0x7f604415d0a0 0x7f60440ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.986+0000 7f604bb70640 1 -- 172.21.15.16:0/3027277471 >> 172.21.15.16:0/3027277471 conn(0x7f60440cfc50 msgr2=0x7f60440d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.986+0000 7f604bb70640 1 -- 172.21.15.16:0/3027277471 shutdown_connections 2024-04-24T23:49:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.986+0000 7f604bb70640 1 -- 172.21.15.16:0/3027277471 wait complete. 2024-04-24T23:49:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.987+0000 7f604bb70640 1 Processor -- start 2024-04-24T23:49:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.987+0000 7f604bb70640 1 -- start start 2024-04-24T23:49:07.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604bb70640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60440ceca0 0x7f604413d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604bb70640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f60441455f0 0x7f60440b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604bb70640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604415d0a0 0x7f60440b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604bb70640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f604406b7f0 con 0x7f60440ceca0 2024-04-24T23:49:07.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604bb70640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6044160260 con 0x7f604415d0a0 2024-04-24T23:49:07.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604bb70640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f60441459d0 con 0x7f60441455f0 2024-04-24T23:49:07.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604990c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60440ceca0 0x7f604413d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604990c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60440ceca0 0x7f604413d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44244/0 (socket says 172.21.15.16:44244) 2024-04-24T23:49:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604990c640 1 -- 172.21.15.16:0/1678165036 learned_addr learned my addr 172.21.15.16:0/1678165036 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604a10d640 1 --2- 172.21.15.16:0/1678165036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604415d0a0 0x7f60440b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604990c640 1 -- 172.21.15.16:0/1678165036 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f60441455f0 msgr2=0x7f60440b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604990c640 1 --2- 172.21.15.16:0/1678165036 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f60441455f0 0x7f60440b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604990c640 1 -- 172.21.15.16:0/1678165036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604415d0a0 msgr2=0x7f60440b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604990c640 1 --2- 172.21.15.16:0/1678165036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604415d0a0 0x7f60440b44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604990c640 1 -- 172.21.15.16:0/1678165036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60440ae160 con 0x7f60440ceca0 2024-04-24T23:49:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604990c640 1 --2- 172.21.15.16:0/1678165036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60440ceca0 0x7f604413d890 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f604000c4e0 tx=0x7f604000c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f603a7fc640 1 -- 172.21.15.16:0/1678165036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040017070 con 0x7f60440ceca0 2024-04-24T23:49:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f603a7fc640 1 -- 172.21.15.16:0/1678165036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60400035f0 con 0x7f60440ceca0 2024-04-24T23:49:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604bb70640 1 -- 172.21.15.16:0/1678165036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60440ae440 con 0x7f60440ceca0 2024-04-24T23:49:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f603a7fc640 1 -- 172.21.15.16:0/1678165036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60400039e0 con 0x7f60440ceca0 2024-04-24T23:49:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.988+0000 7f604bb70640 1 -- 172.21.15.16:0/1678165036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60440aef20 con 0x7f60440ceca0 2024-04-24T23:49:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.990+0000 7f604bb70640 1 -- 172.21.15.16:0/1678165036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6044068f90 con 0x7f60440ceca0 2024-04-24T23:49:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.990+0000 7f603a7fc640 1 -- 172.21.15.16:0/1678165036 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f604001a070 con 0x7f60440ceca0 2024-04-24T23:49:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.990+0000 7f603a7fc640 1 --2- 172.21.15.16:0/1678165036 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f602806be30 0x7f602806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.990+0000 7f603a7fc640 1 -- 172.21.15.16:0/1678165036 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f604008a470 con 0x7f60440ceca0 2024-04-24T23:49:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.991+0000 7f604910b640 1 --2- 172.21.15.16:0/1678165036 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f602806be30 0x7f602806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.991+0000 7f604910b640 1 --2- 172.21.15.16:0/1678165036 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f602806be30 0x7f602806e2f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6034006a90 tx=0x7f6034006a20 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:07.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:07.994+0000 7f603a7fc640 1 -- 172.21.15.16:0/1678165036 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6040058490 con 0x7f60440ceca0 2024-04-24T23:49:08.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.176+0000 7f604bb70640 1 -- 172.21.15.16:0/1678165036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f60440b3150 con 0x7f60440ceca0 2024-04-24T23:49:08.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.177+0000 7f603a7fc640 1 -- 172.21.15.16:0/1678165036 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f604005c790 con 0x7f60440ceca0 2024-04-24T23:49:08.177 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.179+0000 7f604bb70640 1 -- 172.21.15.16:0/1678165036 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f602806be30 msgr2=0x7f602806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.179+0000 7f604bb70640 1 --2- 172.21.15.16:0/1678165036 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f602806be30 0x7f602806e2f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6034006a90 tx=0x7f6034006a20 comp rx=0 tx=0).stop 2024-04-24T23:49:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.179+0000 7f604bb70640 1 -- 172.21.15.16:0/1678165036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60440ceca0 msgr2=0x7f604413d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.179+0000 7f604bb70640 1 --2- 172.21.15.16:0/1678165036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60440ceca0 0x7f604413d890 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f604000c4e0 tx=0x7f604000c9a0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.179+0000 7f604bb70640 1 -- 172.21.15.16:0/1678165036 shutdown_connections 2024-04-24T23:49:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.179+0000 7f604bb70640 1 --2- 172.21.15.16:0/1678165036 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f602806be30 0x7f602806e2f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.179+0000 7f604bb70640 1 --2- 172.21.15.16:0/1678165036 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f604415d0a0 0x7f60440b44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.179+0000 7f604bb70640 1 --2- 172.21.15.16:0/1678165036 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f60441455f0 0x7f60440b3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.179+0000 7f604bb70640 1 --2- 172.21.15.16:0/1678165036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60440ceca0 0x7f604413d890 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.179+0000 7f604bb70640 1 -- 172.21.15.16:0/1678165036 >> 172.21.15.16:0/1678165036 conn(0x7f60440cfc50 msgr2=0x7f60441432a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:08.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.179+0000 7f604bb70640 1 -- 172.21.15.16:0/1678165036 shutdown_connections 2024-04-24T23:49:08.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.179+0000 7f604bb70640 1 -- 172.21.15.16:0/1678165036 wait complete. 2024-04-24T23:49:08.189 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:08.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.293+0000 7f0894d9e640 1 Processor -- start 2024-04-24T23:49:08.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.293+0000 7f0894d9e640 1 -- start start 2024-04-24T23:49:08.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.294+0000 7f0894d9e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f089015dd60 0x7f08900ce920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.294+0000 7f0894d9e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08900d28b0 con 0x7f08900cfa00 2024-04-24T23:49:08.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.294+0000 7f0894d9e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08900d2b90 con 0x7f089015e140 2024-04-24T23:49:08.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.294+0000 7f0894d9e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f08900d2e70 con 0x7f089015dd60 2024-04-24T23:49:08.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.294+0000 7f088ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f08900cfa00 0x7f08900b7090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49682/0 (socket says 172.21.15.16:49682) 2024-04-24T23:49:08.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.294+0000 7f088ffff640 1 -- 172.21.15.16:0/741762308 learned_addr learned my addr 172.21.15.16:0/741762308 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:08.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.294+0000 7f088f7fe640 1 --2- 172.21.15.16:0/741762308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f089015dd60 0x7f08900ce920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.294+0000 7f088e7fc640 1 -- 172.21.15.16:0/741762308 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1841780644 0 0) 0x7f08900d28b0 con 0x7f08900cfa00 2024-04-24T23:49:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.294+0000 7f088e7fc640 1 -- 172.21.15.16:0/741762308 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0878003520 con 0x7f08900cfa00 2024-04-24T23:49:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.294+0000 7f088e7fc640 1 -- 172.21.15.16:0/741762308 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4048171630 0 0) 0x7f0878003520 con 0x7f08900cfa00 2024-04-24T23:49:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.294+0000 7f088e7fc640 1 -- 172.21.15.16:0/741762308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f089015dd60 msgr2=0x7f08900ce920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.294+0000 7f088e7fc640 1 --2- 172.21.15.16:0/741762308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f089015dd60 0x7f08900ce920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.294+0000 7f088e7fc640 1 -- 172.21.15.16:0/741762308 >> v1:172.21.15.156:6789/0 conn(0x7f089015e140 legacy=0x7f08900cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088e7fc640 1 -- 172.21.15.16:0/741762308 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08900d3150 con 0x7f08900cfa00 2024-04-24T23:49:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088e7fc640 1 -- 172.21.15.16:0/741762308 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f08880019f0 con 0x7f08900cfa00 2024-04-24T23:49:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088e7fc640 1 -- 172.21.15.16:0/741762308 >> v1:172.21.15.16:6789/0 conn(0x7f08900cfa00 legacy=0x7f08900b7090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088e7fc640 1 --2- 172.21.15.16:0/741762308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f089015dd60 0x7f08780039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088e7fc640 1 --2- 172.21.15.16:0/741762308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0878003f20 0x7f0878004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088e7fc640 1 --2- 172.21.15.16:0/741762308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0878004d40 0x7f08780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088e7fc640 1 -- 172.21.15.16:0/741762308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08900d2e70 con 0x7f0878003f20 2024-04-24T23:49:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088e7fc640 1 -- 172.21.15.16:0/741762308 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f087800ba20 con 0x7f0878004d40 2024-04-24T23:49:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088e7fc640 1 -- 172.21.15.16:0/741762308 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f087800bc20 con 0x7f089015dd60 2024-04-24T23:49:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088effd640 1 --2- 172.21.15.16:0/741762308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0878003f20 0x7f0878004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088f7fe640 1 --2- 172.21.15.16:0/741762308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f089015dd60 0x7f08780039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088ffff640 1 --2- 172.21.15.16:0/741762308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0878004d40 0x7f08780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088effd640 1 -- 172.21.15.16:0/741762308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f089015dd60 msgr2=0x7f08780039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088effd640 1 --2- 172.21.15.16:0/741762308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f089015dd60 0x7f08780039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088effd640 1 -- 172.21.15.16:0/741762308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0878004d40 msgr2=0x7f08780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088effd640 1 --2- 172.21.15.16:0/741762308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0878004d40 0x7f08780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088effd640 1 -- 172.21.15.16:0/741762308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0878003520 con 0x7f0878003f20 2024-04-24T23:49:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.295+0000 7f088f7fe640 1 --2- 172.21.15.16:0/741762308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f089015dd60 0x7f08780039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.296+0000 7f088effd640 1 --2- 172.21.15.16:0/741762308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0878003f20 0x7f0878004800 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f08840018e0 tx=0x7f0878005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e63fef594610cf1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.296+0000 7f088e7fc640 1 -- 172.21.15.16:0/741762308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884037070 con 0x7f0878003f20 2024-04-24T23:49:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.296+0000 7f088e7fc640 1 -- 172.21.15.16:0/741762308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f088402ab50 con 0x7f0878003f20 2024-04-24T23:49:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.296+0000 7f088e7fc640 1 -- 172.21.15.16:0/741762308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08840014e0 con 0x7f0878003f20 2024-04-24T23:49:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.296+0000 7f0894d9e640 1 -- 172.21.15.16:0/741762308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0878003f20 msgr2=0x7f0878004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.296+0000 7f0894d9e640 1 --2- 172.21.15.16:0/741762308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0878003f20 0x7f0878004800 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f08840018e0 tx=0x7f0878005830 comp rx=0 tx=0).stop 2024-04-24T23:49:08.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.296+0000 7f088ffff640 1 -- 172.21.15.16:0/741762308 reap_dead start 2024-04-24T23:49:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.296+0000 7f0894d9e640 1 -- 172.21.15.16:0/741762308 shutdown_connections 2024-04-24T23:49:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.296+0000 7f0894d9e640 1 -- 172.21.15.16:0/741762308 >> 172.21.15.16:0/741762308 conn(0x7f08900d4000 msgr2=0x7f08900d4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.297+0000 7f0894d9e640 1 -- 172.21.15.16:0/741762308 shutdown_connections 2024-04-24T23:49:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.297+0000 7f0894d9e640 1 -- 172.21.15.16:0/741762308 wait complete. 2024-04-24T23:49:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.298+0000 7f0894d9e640 1 Processor -- start 2024-04-24T23:49:08.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.298+0000 7f0894d9e640 1 -- start start 2024-04-24T23:49:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f0894d9e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0878003f20 0x7f08901560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f0894d9e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08900ccf40 0x7f08900b27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f0894d9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08900b2d30 0x7f0890154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f0894d9e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f08900d2670 con 0x7f08900b2d30 2024-04-24T23:49:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f0894d9e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f08900d24f0 con 0x7f08900ccf40 2024-04-24T23:49:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f0894d9e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f08900b31f0 con 0x7f0878003f20 2024-04-24T23:49:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f088ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08900b2d30 0x7f0890154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f088ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08900b2d30 0x7f0890154d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44270/0 (socket says 172.21.15.16:44270) 2024-04-24T23:49:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f088ffff640 1 -- 172.21.15.16:0/2469599680 learned_addr learned my addr 172.21.15.16:0/2469599680 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f088effd640 1 --2- 172.21.15.16:0/2469599680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08900ccf40 0x7f08900b27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f088f7fe640 1 --2- 172.21.15.16:0/2469599680 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0878003f20 0x7f08901560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f088ffff640 1 -- 172.21.15.16:0/2469599680 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0878003f20 msgr2=0x7f08901560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f088ffff640 1 --2- 172.21.15.16:0/2469599680 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0878003f20 0x7f08901560c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f088ffff640 1 -- 172.21.15.16:0/2469599680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08900ccf40 msgr2=0x7f08900b27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f088ffff640 1 --2- 172.21.15.16:0/2469599680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08900ccf40 0x7f08900b27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f088ffff640 1 -- 172.21.15.16:0/2469599680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0890153cf0 con 0x7f08900b2d30 2024-04-24T23:49:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f088ffff640 1 --2- 172.21.15.16:0/2469599680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08900b2d30 0x7f0890154d20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f08900ce870 tx=0x7f0888007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f0877fff640 1 -- 172.21.15.16:0/2469599680 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0880009690 con 0x7f08900b2d30 2024-04-24T23:49:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f0877fff640 1 -- 172.21.15.16:0/2469599680 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0888018070 con 0x7f08900b2d30 2024-04-24T23:49:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.299+0000 7f0894d9e640 1 -- 172.21.15.16:0/2469599680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0890153f60 con 0x7f08900b2d30 2024-04-24T23:49:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.300+0000 7f0877fff640 1 -- 172.21.15.16:0/2469599680 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088800ad50 con 0x7f08900b2d30 2024-04-24T23:49:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.300+0000 7f0894d9e640 1 -- 172.21.15.16:0/2469599680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0890154800 con 0x7f08900b2d30 2024-04-24T23:49:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.300+0000 7f0894d9e640 1 -- 172.21.15.16:0/2469599680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0890154130 con 0x7f08900b2d30 2024-04-24T23:49:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.301+0000 7f0877fff640 1 -- 172.21.15.16:0/2469599680 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f088801d460 con 0x7f08900b2d30 2024-04-24T23:49:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.301+0000 7f0877fff640 1 --2- 172.21.15.16:0/2469599680 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f087c06be30 0x7f087c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.301+0000 7f0877fff640 1 -- 172.21.15.16:0/2469599680 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f088808b660 con 0x7f08900b2d30 2024-04-24T23:49:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.301+0000 7f088f7fe640 1 --2- 172.21.15.16:0/2469599680 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f087c06be30 0x7f087c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.302+0000 7f088f7fe640 1 --2- 172.21.15.16:0/2469599680 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f087c06be30 0x7f087c06e2f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0890153c00 tx=0x7f0880008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:08.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.306+0000 7f0877fff640 1 -- 172.21.15.16:0/2469599680 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0888059c30 con 0x7f08900b2d30 2024-04-24T23:49:08.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.492+0000 7f0894d9e640 1 -- 172.21.15.16:0/2469599680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0890154a00 con 0x7f08900b2d30 2024-04-24T23:49:08.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.492+0000 7f0877fff640 1 -- 172.21.15.16:0/2469599680 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f0890154a00 con 0x7f08900b2d30 2024-04-24T23:49:08.492 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:08.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.494+0000 7f0894d9e640 1 -- 172.21.15.16:0/2469599680 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f087c06be30 msgr2=0x7f087c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.494+0000 7f0894d9e640 1 --2- 172.21.15.16:0/2469599680 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f087c06be30 0x7f087c06e2f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0890153c00 tx=0x7f0880008040 comp rx=0 tx=0).stop 2024-04-24T23:49:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.494+0000 7f0894d9e640 1 -- 172.21.15.16:0/2469599680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08900b2d30 msgr2=0x7f0890154d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.494+0000 7f0894d9e640 1 --2- 172.21.15.16:0/2469599680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08900b2d30 0x7f0890154d20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f08900ce870 tx=0x7f0888007bd0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.494+0000 7f0894d9e640 1 -- 172.21.15.16:0/2469599680 shutdown_connections 2024-04-24T23:49:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.494+0000 7f0894d9e640 1 --2- 172.21.15.16:0/2469599680 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f087c06be30 0x7f087c06e2f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.494+0000 7f0894d9e640 1 --2- 172.21.15.16:0/2469599680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08900b2d30 0x7f0890154d20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.494+0000 7f0894d9e640 1 --2- 172.21.15.16:0/2469599680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f08900ccf40 0x7f08900b27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.494+0000 7f0894d9e640 1 --2- 172.21.15.16:0/2469599680 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0878003f20 0x7f08901560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.494+0000 7f0894d9e640 1 -- 172.21.15.16:0/2469599680 >> 172.21.15.16:0/2469599680 conn(0x7f08900d4000 msgr2=0x7f08900b7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.494+0000 7f0894d9e640 1 -- 172.21.15.16:0/2469599680 shutdown_connections 2024-04-24T23:49:08.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.494+0000 7f0894d9e640 1 -- 172.21.15.16:0/2469599680 wait complete. 2024-04-24T23:49:08.505 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.609+0000 7fe878929640 1 Processor -- start 2024-04-24T23:49:08.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.609+0000 7fe878929640 1 -- start start 2024-04-24T23:49:08.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.610+0000 7fe878929640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8740d2840 0x7fe8740d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.610+0000 7fe878929640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87415ec30 con 0x7fe8740d12d0 2024-04-24T23:49:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.610+0000 7fe878929640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87415ef10 con 0x7fe8740cd790 2024-04-24T23:49:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.610+0000 7fe878929640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe87415f4d0 con 0x7fe8740d2840 2024-04-24T23:49:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.610+0000 7fe87259c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe8740d12d0 0x7fe8740c8580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49696/0 (socket says 172.21.15.16:49696) 2024-04-24T23:49:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.610+0000 7fe87259c640 1 -- 172.21.15.16:0/2943038069 learned_addr learned my addr 172.21.15.16:0/2943038069 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.610+0000 7fe87159a640 1 -- 172.21.15.16:0/2943038069 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3265363006 0 0) 0x7fe87415ec30 con 0x7fe8740d12d0 2024-04-24T23:49:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87159a640 1 -- 172.21.15.16:0/2943038069 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe860003520 con 0x7fe8740d12d0 2024-04-24T23:49:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe872d9d640 1 --2- 172.21.15.16:0/2943038069 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8740d2840 0x7fe8740d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87159a640 1 -- 172.21.15.16:0/2943038069 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3451244266 0 0) 0x7fe860003520 con 0x7fe8740d12d0 2024-04-24T23:49:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87159a640 1 -- 172.21.15.16:0/2943038069 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8740d2840 msgr2=0x7fe8740d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87159a640 1 --2- 172.21.15.16:0/2943038069 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8740d2840 0x7fe8740d16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87159a640 1 -- 172.21.15.16:0/2943038069 >> v1:172.21.15.156:6789/0 conn(0x7fe8740cd790 legacy=0x7fe8740d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87159a640 1 -- 172.21.15.16:0/2943038069 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87415f1f0 con 0x7fe8740d12d0 2024-04-24T23:49:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87159a640 1 -- 172.21.15.16:0/2943038069 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7fe86c001670 con 0x7fe8740d12d0 2024-04-24T23:49:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87159a640 1 -- 172.21.15.16:0/2943038069 >> v1:172.21.15.16:6789/0 conn(0x7fe8740d12d0 legacy=0x7fe8740c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87159a640 1 --2- 172.21.15.16:0/2943038069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8600039e0 0x7fe860003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87159a640 1 --2- 172.21.15.16:0/2943038069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe860004800 0x7fe860004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87159a640 1 --2- 172.21.15.16:0/2943038069 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8740d2840 0x7fe860005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87159a640 1 -- 172.21.15.16:0/2943038069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe87415f4d0 con 0x7fe860004800 2024-04-24T23:49:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87159a640 1 -- 172.21.15.16:0/2943038069 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe86000bbb0 con 0x7fe8600039e0 2024-04-24T23:49:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87159a640 1 -- 172.21.15.16:0/2943038069 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe86000bdb0 con 0x7fe8740d2840 2024-04-24T23:49:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe871d9b640 1 --2- 172.21.15.16:0/2943038069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe860004800 0x7fe860004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe87259c640 1 --2- 172.21.15.16:0/2943038069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8600039e0 0x7fe860003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe872d9d640 1 --2- 172.21.15.16:0/2943038069 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8740d2840 0x7fe860005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe871d9b640 1 -- 172.21.15.16:0/2943038069 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8740d2840 msgr2=0x7fe860005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe871d9b640 1 --2- 172.21.15.16:0/2943038069 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe8740d2840 0x7fe860005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe871d9b640 1 -- 172.21.15.16:0/2943038069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8600039e0 msgr2=0x7fe860003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe871d9b640 1 --2- 172.21.15.16:0/2943038069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8600039e0 0x7fe860003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.611+0000 7fe871d9b640 1 -- 172.21.15.16:0/2943038069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe860003520 con 0x7fe860004800 2024-04-24T23:49:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.612+0000 7fe871d9b640 1 --2- 172.21.15.16:0/2943038069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe860004800 0x7fe860004c80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe85c001750 tx=0x7fe860005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7be667ccb8218c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.612+0000 7fe87159a640 1 -- 172.21.15.16:0/2943038069 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c001370 con 0x7fe860004800 2024-04-24T23:49:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.612+0000 7fe87159a640 1 -- 172.21.15.16:0/2943038069 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe85c0052d0 con 0x7fe860004800 2024-04-24T23:49:08.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.612+0000 7fe87159a640 1 -- 172.21.15.16:0/2943038069 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85c02abe0 con 0x7fe860004800 2024-04-24T23:49:08.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.613+0000 7fe878929640 1 -- 172.21.15.16:0/2943038069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe860004800 msgr2=0x7fe860004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.613+0000 7fe878929640 1 --2- 172.21.15.16:0/2943038069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe860004800 0x7fe860004c80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe85c001750 tx=0x7fe860005830 comp rx=0 tx=0).stop 2024-04-24T23:49:08.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.613+0000 7fe872d9d640 1 -- 172.21.15.16:0/2943038069 reap_dead start 2024-04-24T23:49:08.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.613+0000 7fe878929640 1 -- 172.21.15.16:0/2943038069 shutdown_connections 2024-04-24T23:49:08.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.613+0000 7fe878929640 1 -- 172.21.15.16:0/2943038069 >> 172.21.15.16:0/2943038069 conn(0x7fe874072e70 msgr2=0x7fe874073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:08.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.613+0000 7fe878929640 1 -- 172.21.15.16:0/2943038069 shutdown_connections 2024-04-24T23:49:08.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.613+0000 7fe878929640 1 -- 172.21.15.16:0/2943038069 wait complete. 2024-04-24T23:49:08.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.614+0000 7fe878929640 1 Processor -- start 2024-04-24T23:49:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.614+0000 7fe878929640 1 -- start start 2024-04-24T23:49:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe878929640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe860004800 0x7fe874148480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe878929640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe874155550 0x7fe8741559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe878929640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8741576e0 0x7fe874157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe878929640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe8740d1830 con 0x7fe860004800 2024-04-24T23:49:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe878929640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe8740d16b0 con 0x7fe8741576e0 2024-04-24T23:49:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe878929640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe8740d19b0 con 0x7fe874155550 2024-04-24T23:49:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe87259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe860004800 0x7fe874148480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe87259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe860004800 0x7fe874148480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44294/0 (socket says 172.21.15.16:44294) 2024-04-24T23:49:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe87259c640 1 -- 172.21.15.16:0/578335128 learned_addr learned my addr 172.21.15.16:0/578335128 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe871d9b640 1 --2- 172.21.15.16:0/578335128 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe874155550 0x7fe8741559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe872d9d640 1 --2- 172.21.15.16:0/578335128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8741576e0 0x7fe874157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe87259c640 1 -- 172.21.15.16:0/578335128 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe874155550 msgr2=0x7fe8741559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe87259c640 1 --2- 172.21.15.16:0/578335128 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe874155550 0x7fe8741559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe87259c640 1 -- 172.21.15.16:0/578335128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8741576e0 msgr2=0x7fe874157b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe87259c640 1 --2- 172.21.15.16:0/578335128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8741576e0 0x7fe874157b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe87259c640 1 -- 172.21.15.16:0/578335128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe874158410 con 0x7fe860004800 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe87259c640 1 --2- 172.21.15.16:0/578335128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe860004800 0x7fe874148480 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe86c004ec0 tx=0x7fe86c007430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.615+0000 7fe86affd640 1 -- 172.21.15.16:0/578335128 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c00f7b0 con 0x7fe860004800 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.616+0000 7fe86affd640 1 -- 172.21.15.16:0/578335128 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86c018070 con 0x7fe860004800 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.616+0000 7fe878929640 1 -- 172.21.15.16:0/578335128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8740c9330 con 0x7fe860004800 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.616+0000 7fe86affd640 1 -- 172.21.15.16:0/578335128 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c00fbb0 con 0x7fe860004800 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.616+0000 7fe878929640 1 -- 172.21.15.16:0/578335128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8740c98f0 con 0x7fe860004800 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.616+0000 7fe878929640 1 -- 172.21.15.16:0/578335128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe874068f90 con 0x7fe860004800 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.617+0000 7fe86affd640 1 -- 172.21.15.16:0/578335128 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe86c00fd50 con 0x7fe860004800 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.617+0000 7fe86affd640 1 --2- 172.21.15.16:0/578335128 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe85406be30 0x7fe85406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.617+0000 7fe86affd640 1 -- 172.21.15.16:0/578335128 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe86c08af50 con 0x7fe860004800 2024-04-24T23:49:08.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.618+0000 7fe871d9b640 1 --2- 172.21.15.16:0/578335128 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe85406be30 0x7fe85406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.618+0000 7fe871d9b640 1 --2- 172.21.15.16:0/578335128 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe85406be30 0x7fe85406e2f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe85c032000 tx=0x7fe85c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:08.621 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.622+0000 7fe86affd640 1 -- 172.21.15.16:0/578335128 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe874158410 con 0x7fe860004800 2024-04-24T23:49:08.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.807+0000 7fe878929640 1 -- 172.21.15.16:0/578335128 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe8740c7040 con 0x7fe860004800 2024-04-24T23:49:08.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.808+0000 7fe86affd640 1 -- 172.21.15.16:0/578335128 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7fe86c059000 con 0x7fe860004800 2024-04-24T23:49:08.808 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:08.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.810+0000 7fe878929640 1 -- 172.21.15.16:0/578335128 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe85406be30 msgr2=0x7fe85406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.810+0000 7fe878929640 1 --2- 172.21.15.16:0/578335128 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe85406be30 0x7fe85406e2f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe85c032000 tx=0x7fe85c02e000 comp rx=0 tx=0).stop 2024-04-24T23:49:08.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.810+0000 7fe878929640 1 -- 172.21.15.16:0/578335128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe860004800 msgr2=0x7fe874148480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.810+0000 7fe878929640 1 --2- 172.21.15.16:0/578335128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe860004800 0x7fe874148480 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe86c004ec0 tx=0x7fe86c007430 comp rx=0 tx=0).stop 2024-04-24T23:49:08.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.810+0000 7fe878929640 1 -- 172.21.15.16:0/578335128 shutdown_connections 2024-04-24T23:49:08.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.810+0000 7fe878929640 1 --2- 172.21.15.16:0/578335128 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe85406be30 0x7fe85406e2f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.810+0000 7fe878929640 1 --2- 172.21.15.16:0/578335128 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe8741576e0 0x7fe874157b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.810+0000 7fe878929640 1 --2- 172.21.15.16:0/578335128 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe874155550 0x7fe8741559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.810+0000 7fe878929640 1 --2- 172.21.15.16:0/578335128 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe860004800 0x7fe874148480 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.810+0000 7fe878929640 1 -- 172.21.15.16:0/578335128 >> 172.21.15.16:0/578335128 conn(0x7fe874072e70 msgr2=0x7fe874160ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:08.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.810+0000 7fe878929640 1 -- 172.21.15.16:0/578335128 shutdown_connections 2024-04-24T23:49:08.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.810+0000 7fe878929640 1 -- 172.21.15.16:0/578335128 wait complete. 2024-04-24T23:49:08.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.923+0000 7f53d8bc1640 1 Processor -- start 2024-04-24T23:49:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.923+0000 7f53d8bc1640 1 -- start start 2024-04-24T23:49:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.924+0000 7f53d8bc1640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f53d415e2f0 0x7f53d406b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.924+0000 7f53d8bc1640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53d40d26e0 0x7f53d406b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.924+0000 7f53d8bc1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f53d406be90 0x7f53d40cf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.924+0000 7f53d8bc1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f53d415e6d0 con 0x7f53d406be90 2024-04-24T23:49:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.924+0000 7f53d8bc1640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f53d40d2ca0 con 0x7f53d415e2f0 2024-04-24T23:49:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.924+0000 7f53d8bc1640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f53d40d2f80 con 0x7f53d40d26e0 2024-04-24T23:49:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.924+0000 7f53d2d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f53d406be90 0x7f53d40cf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.924+0000 7f53d2d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f53d406be90 0x7f53d40cf070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44308/0 (socket says 172.21.15.16:44308) 2024-04-24T23:49:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.924+0000 7f53d2d9d640 1 -- 172.21.15.16:0/1095230762 learned_addr learned my addr 172.21.15.16:0/1095230762 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.924+0000 7f53d259c640 1 --2- 172.21.15.16:0/1095230762 >> v2:172.21.15.156:3300/0 conn(0x7f53d415e2f0 0x7f53d406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.924+0000 7f53d1d9b640 1 --2- 172.21.15.16:0/1095230762 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53d40d26e0 0x7f53d406b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.925+0000 7f53d2d9d640 1 -- 172.21.15.16:0/1095230762 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53d40d26e0 msgr2=0x7f53d406b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.925+0000 7f53d2d9d640 1 --2- 172.21.15.16:0/1095230762 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53d40d26e0 0x7f53d406b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.925+0000 7f53d2d9d640 1 -- 172.21.15.16:0/1095230762 >> v2:172.21.15.156:3300/0 conn(0x7f53d415e2f0 msgr2=0x7f53d406b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.925+0000 7f53d2d9d640 1 --2- 172.21.15.16:0/1095230762 >> v2:172.21.15.156:3300/0 conn(0x7f53d415e2f0 0x7f53d406b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.925+0000 7f53d2d9d640 1 -- 172.21.15.16:0/1095230762 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d40d3260 con 0x7f53d406be90 2024-04-24T23:49:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.925+0000 7f53d2d9d640 1 --2- 172.21.15.16:0/1095230762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53d406be90 0x7f53d40cf070 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f53d40d1500 tx=0x7f53c402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0e2b15da2569ed server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.925+0000 7f53d159a640 1 -- 172.21.15.16:0/1095230762 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4036070 con 0x7f53d406be90 2024-04-24T23:49:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.925+0000 7f53d159a640 1 -- 172.21.15.16:0/1095230762 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c4031600 con 0x7f53d406be90 2024-04-24T23:49:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.925+0000 7f53d159a640 1 -- 172.21.15.16:0/1095230762 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c40319d0 con 0x7f53d406be90 2024-04-24T23:49:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.926+0000 7f53d8bc1640 1 -- 172.21.15.16:0/1095230762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53d406be90 msgr2=0x7f53d40cf070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.926+0000 7f53d8bc1640 1 --2- 172.21.15.16:0/1095230762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53d406be90 0x7f53d40cf070 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f53d40d1500 tx=0x7f53c402a730 comp rx=0 tx=0).stop 2024-04-24T23:49:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.926+0000 7f53d8bc1640 1 -- 172.21.15.16:0/1095230762 shutdown_connections 2024-04-24T23:49:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.926+0000 7f53d8bc1640 1 --2- 172.21.15.16:0/1095230762 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53d406be90 0x7f53d40cf070 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.926+0000 7f53d8bc1640 1 --2- 172.21.15.16:0/1095230762 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53d40d26e0 0x7f53d406b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.926+0000 7f53d8bc1640 1 --2- 172.21.15.16:0/1095230762 >> v2:172.21.15.156:3300/0 conn(0x7f53d415e2f0 0x7f53d406b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.926+0000 7f53d8bc1640 1 -- 172.21.15.16:0/1095230762 >> 172.21.15.16:0/1095230762 conn(0x7f53d40d4850 msgr2=0x7f53d40d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.926+0000 7f53d8bc1640 1 -- 172.21.15.16:0/1095230762 shutdown_connections 2024-04-24T23:49:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.926+0000 7f53d8bc1640 1 -- 172.21.15.16:0/1095230762 wait complete. 2024-04-24T23:49:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.927+0000 7f53d8bc1640 1 Processor -- start 2024-04-24T23:49:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.927+0000 7f53d8bc1640 1 -- start start 2024-04-24T23:49:08.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.928+0000 7f53d8bc1640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53d406be90 0x7f53d414dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.928+0000 7f53d8bc1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53d40d26e0 0x7f53d4161160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.928+0000 7f53d8bc1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53d415e2f0 0x7f53d40cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.928+0000 7f53d8bc1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f53d40cf5b0 con 0x7f53d415e2f0 2024-04-24T23:49:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.928+0000 7f53d8bc1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f53d40b70a0 con 0x7f53d40d26e0 2024-04-24T23:49:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.928+0000 7f53d8bc1640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f53d40cf730 con 0x7f53d406be90 2024-04-24T23:49:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.928+0000 7f53d2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53d415e2f0 0x7f53d40cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.928+0000 7f53d2d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53d415e2f0 0x7f53d40cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44316/0 (socket says 172.21.15.16:44316) 2024-04-24T23:49:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.928+0000 7f53d2d9d640 1 -- 172.21.15.16:0/2669039862 learned_addr learned my addr 172.21.15.16:0/2669039862 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.928+0000 7f53d1d9b640 1 --2- 172.21.15.16:0/2669039862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53d40d26e0 0x7f53d4161160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.928+0000 7f53d259c640 1 --2- 172.21.15.16:0/2669039862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53d406be90 0x7f53d414dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.929+0000 7f53d2d9d640 1 -- 172.21.15.16:0/2669039862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53d406be90 msgr2=0x7f53d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.929+0000 7f53d2d9d640 1 --2- 172.21.15.16:0/2669039862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53d406be90 0x7f53d414dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.929+0000 7f53d2d9d640 1 -- 172.21.15.16:0/2669039862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53d40d26e0 msgr2=0x7f53d4161160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.929+0000 7f53d2d9d640 1 --2- 172.21.15.16:0/2669039862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53d40d26e0 0x7f53d4161160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.929+0000 7f53d2d9d640 1 -- 172.21.15.16:0/2669039862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53d40cc330 con 0x7f53d415e2f0 2024-04-24T23:49:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.929+0000 7f53d2d9d640 1 --2- 172.21.15.16:0/2669039862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53d415e2f0 0x7f53d40cd3d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009f40 tx=0x7f53c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.929+0000 7f53c2ffd640 1 -- 172.21.15.16:0/2669039862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4036050 con 0x7f53d415e2f0 2024-04-24T23:49:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.929+0000 7f53c2ffd640 1 -- 172.21.15.16:0/2669039862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53c4042070 con 0x7f53d415e2f0 2024-04-24T23:49:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.929+0000 7f53d8bc1640 1 -- 172.21.15.16:0/2669039862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53d40cc610 con 0x7f53d415e2f0 2024-04-24T23:49:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.929+0000 7f53c2ffd640 1 -- 172.21.15.16:0/2669039862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c4009510 con 0x7f53d415e2f0 2024-04-24T23:49:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.929+0000 7f53d8bc1640 1 -- 172.21.15.16:0/2669039862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53d40cceb0 con 0x7f53d415e2f0 2024-04-24T23:49:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.930+0000 7f53d8bc1640 1 -- 172.21.15.16:0/2669039862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53d40cca00 con 0x7f53d415e2f0 2024-04-24T23:49:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.931+0000 7f53c2ffd640 1 -- 172.21.15.16:0/2669039862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f53c4007bf0 con 0x7f53d415e2f0 2024-04-24T23:49:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.931+0000 7f53c2ffd640 1 --2- 172.21.15.16:0/2669039862 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f53b406be30 0x7f53b406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.931+0000 7f53c2ffd640 1 -- 172.21.15.16:0/2669039862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f53c40abe90 con 0x7f53d415e2f0 2024-04-24T23:49:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.931+0000 7f53d259c640 1 --2- 172.21.15.16:0/2669039862 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f53b406be30 0x7f53b406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.932+0000 7f53d259c640 1 --2- 172.21.15.16:0/2669039862 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f53b406be30 0x7f53b406e2f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f53cc0079a0 tx=0x7f53cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:08.935+0000 7f53c2ffd640 1 -- 172.21.15.16:0/2669039862 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f53c4079eb0 con 0x7f53d415e2f0 2024-04-24T23:49:09.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.116+0000 7f53d8bc1640 1 -- 172.21.15.16:0/2669039862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f53d4068f90 con 0x7f53d415e2f0 2024-04-24T23:49:09.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.116+0000 7f53c2ffd640 1 -- 172.21.15.16:0/2669039862 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7f53c407e1b0 con 0x7f53d415e2f0 2024-04-24T23:49:09.116 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:09.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.118+0000 7f53d8bc1640 1 -- 172.21.15.16:0/2669039862 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f53b406be30 msgr2=0x7f53b406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:09.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.118+0000 7f53d8bc1640 1 --2- 172.21.15.16:0/2669039862 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f53b406be30 0x7f53b406e2f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f53cc0079a0 tx=0x7f53cc008040 comp rx=0 tx=0).stop 2024-04-24T23:49:09.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.118+0000 7f53d8bc1640 1 -- 172.21.15.16:0/2669039862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53d415e2f0 msgr2=0x7f53d40cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:09.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.118+0000 7f53d8bc1640 1 --2- 172.21.15.16:0/2669039862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53d415e2f0 0x7f53d40cd3d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009f40 tx=0x7f53c4004a40 comp rx=0 tx=0).stop 2024-04-24T23:49:09.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.118+0000 7f53d8bc1640 1 -- 172.21.15.16:0/2669039862 shutdown_connections 2024-04-24T23:49:09.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.118+0000 7f53d8bc1640 1 --2- 172.21.15.16:0/2669039862 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f53b406be30 0x7f53b406e2f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:09.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.118+0000 7f53d8bc1640 1 --2- 172.21.15.16:0/2669039862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53d415e2f0 0x7f53d40cd3d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:09.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.118+0000 7f53d8bc1640 1 --2- 172.21.15.16:0/2669039862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f53d40d26e0 0x7f53d4161160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:09.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.118+0000 7f53d8bc1640 1 --2- 172.21.15.16:0/2669039862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f53d406be90 0x7f53d414dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:09.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.118+0000 7f53d8bc1640 1 -- 172.21.15.16:0/2669039862 >> 172.21.15.16:0/2669039862 conn(0x7f53d40d4850 msgr2=0x7f53d40b8750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:09.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.118+0000 7f53d8bc1640 1 -- 172.21.15.16:0/2669039862 shutdown_connections 2024-04-24T23:49:09.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.118+0000 7f53d8bc1640 1 -- 172.21.15.16:0/2669039862 wait complete. 2024-04-24T23:49:09.129 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:49:09.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.231+0000 7ffb6055a640 1 Processor -- start 2024-04-24T23:49:09.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.231+0000 7ffb6055a640 1 -- start start 2024-04-24T23:49:09.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.232+0000 7ffb6055a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb5815e930 0x7ffb5815ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.232+0000 7ffb6055a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ffb580cff10 0x7ffb5806b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.232+0000 7ffb6055a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb580d0370 con 0x7ffb5815dac0 2024-04-24T23:49:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.232+0000 7ffb6055a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ffb5806bcd0 con 0x7ffb580cff10 2024-04-24T23:49:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.232+0000 7ffb6055a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ffb5806bfb0 con 0x7ffb5815e930 2024-04-24T23:49:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.232+0000 7ffb5daf5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ffb580cff10 0x7ffb5806b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.232+0000 7ffb5e2f6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb5815e930 0x7ffb5815ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.232+0000 7ffb5daf5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ffb580cff10 0x7ffb5806b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:33368/0 (socket says 172.21.15.16:33368) 2024-04-24T23:49:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.232+0000 7ffb5eaf7640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7ffb5815dac0 0x7ffb5815df20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:52172/0 (socket says 172.21.15.16:52172) 2024-04-24T23:49:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.232+0000 7ffb5daf5640 1 -- 172.21.15.16:0/3438409612 learned_addr learned my addr 172.21.15.16:0/3438409612 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.232+0000 7ffb5e2f6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb5815e930 0x7ffb5815ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:37462/0 (socket says 172.21.15.16:37462) 2024-04-24T23:49:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.233+0000 7ffb5daf5640 1 -- 172.21.15.16:0/3438409612 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb5815e930 msgr2=0x7ffb5815ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.233+0000 7ffb5daf5640 1 --2- 172.21.15.16:0/3438409612 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb5815e930 0x7ffb5815ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:09.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.233+0000 7ffb5daf5640 1 -- 172.21.15.16:0/3438409612 >> v1:172.21.15.156:6789/0 conn(0x7ffb5815dac0 legacy=0x7ffb5815df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.233+0000 7ffb5d2f4640 1 -- 172.21.15.16:0/3438409612 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3185037012 0 0) 0x7ffb48001490 con 0x7ffb5815dac0 2024-04-24T23:49:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.233+0000 7ffb5daf5640 1 -- 172.21.15.16:0/3438409612 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb5806c290 con 0x7ffb580cff10 2024-04-24T23:49:09.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.233+0000 7ffb5e2f6640 1 --2- 172.21.15.16:0/3438409612 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb5815e930 0x7ffb5815ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.234+0000 7ffb5daf5640 1 --2- 172.21.15.16:0/3438409612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb580cff10 0x7ffb5806b5a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009870 tx=0x7ffb4c02a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25d878e9b6731a1d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.234+0000 7ffb5d2f4640 1 -- 172.21.15.16:0/3438409612 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c037070 con 0x7ffb580cff10 2024-04-24T23:49:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.234+0000 7ffb5d2f4640 1 -- 172.21.15.16:0/3438409612 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb4c032440 con 0x7ffb580cff10 2024-04-24T23:49:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.234+0000 7ffb5d2f4640 1 -- 172.21.15.16:0/3438409612 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb4c032810 con 0x7ffb580cff10 2024-04-24T23:49:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.234+0000 7ffb6055a640 1 -- 172.21.15.16:0/3438409612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb580cff10 msgr2=0x7ffb5806b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:09.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.234+0000 7ffb6055a640 1 --2- 172.21.15.16:0/3438409612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb580cff10 0x7ffb5806b5a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c009870 tx=0x7ffb4c02a270 comp rx=0 tx=0).stop 2024-04-24T23:49:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.234+0000 7ffb6055a640 1 -- 172.21.15.16:0/3438409612 shutdown_connections 2024-04-24T23:49:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.234+0000 7ffb6055a640 1 --2- 172.21.15.16:0/3438409612 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb580cff10 0x7ffb5806b5a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.234+0000 7ffb6055a640 1 --2- 172.21.15.16:0/3438409612 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb5815e930 0x7ffb5815ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.234+0000 7ffb6055a640 1 -- 172.21.15.16:0/3438409612 >> 172.21.15.16:0/3438409612 conn(0x7ffb580d4050 msgr2=0x7ffb580d4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.234+0000 7ffb6055a640 1 -- 172.21.15.16:0/3438409612 shutdown_connections 2024-04-24T23:49:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.234+0000 7ffb6055a640 1 -- 172.21.15.16:0/3438409612 wait complete. 2024-04-24T23:49:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.236+0000 7ffb6055a640 1 Processor -- start 2024-04-24T23:49:09.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.236+0000 7ffb6055a640 1 -- start start 2024-04-24T23:49:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.236+0000 7ffb6055a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb580cff10 0x7ffb580c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.236+0000 7ffb6055a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb5815dac0 0x7ffb5814b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.236+0000 7ffb6055a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb5815e930 0x7ffb581f72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:09.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.236+0000 7ffb6055a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffb580d1bf0 con 0x7ffb580cff10 2024-04-24T23:49:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.236+0000 7ffb6055a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ffb5806c850 con 0x7ffb5815dac0 2024-04-24T23:49:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.236+0000 7ffb6055a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ffb580d1d70 con 0x7ffb5815e930 2024-04-24T23:49:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.236+0000 7ffb5e2f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb580cff10 0x7ffb580c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.236+0000 7ffb5e2f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb580cff10 0x7ffb580c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44326/0 (socket says 172.21.15.16:44326) 2024-04-24T23:49:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb5e2f6640 1 -- 172.21.15.16:0/4037452301 learned_addr learned my addr 172.21.15.16:0/4037452301 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb5daf5640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb5815dac0 0x7ffb5814b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb5eaf7640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb5815e930 0x7ffb581f72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb5e2f6640 1 -- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb5815e930 msgr2=0x7ffb581f72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb5e2f6640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb5815e930 0x7ffb581f72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb5e2f6640 1 -- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb5815dac0 msgr2=0x7ffb5814b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb5e2f6640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb5815dac0 0x7ffb5814b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb5e2f6640 1 -- 172.21.15.16:0/4037452301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb581f3de0 con 0x7ffb580cff10 2024-04-24T23:49:09.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb5daf5640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb5815dac0 0x7ffb5814b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb5e2f6640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb580cff10 0x7ffb580c9bc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000e990 tx=0x7ffb5000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb56ffd640 1 -- 172.21.15.16:0/4037452301 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb50017070 con 0x7ffb580cff10 2024-04-24T23:49:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb56ffd640 1 -- 172.21.15.16:0/4037452301 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb5000c9e0 con 0x7ffb580cff10 2024-04-24T23:49:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb6055a640 1 -- 172.21.15.16:0/4037452301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb581f7af0 con 0x7ffb580cff10 2024-04-24T23:49:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb56ffd640 1 -- 172.21.15.16:0/4037452301 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb5000cdb0 con 0x7ffb580cff10 2024-04-24T23:49:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.237+0000 7ffb6055a640 1 -- 172.21.15.16:0/4037452301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb581f4050 con 0x7ffb580cff10 2024-04-24T23:49:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.238+0000 7ffb6055a640 1 -- 172.21.15.16:0/4037452301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb5806dfe0 con 0x7ffb580cff10 2024-04-24T23:49:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.239+0000 7ffb56ffd640 1 -- 172.21.15.16:0/4037452301 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ffb5001a070 con 0x7ffb580cff10 2024-04-24T23:49:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.239+0000 7ffb56ffd640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffb4406be30 0x7ffb4406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.239+0000 7ffb56ffd640 1 -- 172.21.15.16:0/4037452301 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ffb5008b0a0 con 0x7ffb580cff10 2024-04-24T23:49:09.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.240+0000 7ffb5daf5640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffb4406be30 0x7ffb4406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:09.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.240+0000 7ffb5daf5640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffb4406be30 0x7ffb4406e2f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c034000 tx=0x7ffb4c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:09.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.244+0000 7ffb56ffd640 1 -- 172.21.15.16:0/4037452301 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ffb50059710 con 0x7ffb580cff10 2024-04-24T23:49:09.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.438+0000 7ffb6055a640 1 -- 172.21.15.16:0/4037452301 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ffb58162b60 con 0x7ffb580cff10 2024-04-24T23:49:09.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.438+0000 7ffb56ffd640 1 -- 172.21.15.16:0/4037452301 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) ==== 80+0+0 (secure 0 0 0) 0x7ffb50059010 con 0x7ffb580cff10 2024-04-24T23:49:09.438 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-04-24T23:49:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.440+0000 7ffb6055a640 1 -- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffb4406be30 msgr2=0x7ffb4406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.440+0000 7ffb6055a640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffb4406be30 0x7ffb4406e2f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c034000 tx=0x7ffb4c02e000 comp rx=0 tx=0).stop 2024-04-24T23:49:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.440+0000 7ffb6055a640 1 -- 172.21.15.16:0/4037452301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb580cff10 msgr2=0x7ffb580c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.440+0000 7ffb6055a640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb580cff10 0x7ffb580c9bc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffb5000e990 tx=0x7ffb5000ee50 comp rx=0 tx=0).stop 2024-04-24T23:49:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.440+0000 7ffb6055a640 1 -- 172.21.15.16:0/4037452301 shutdown_connections 2024-04-24T23:49:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.440+0000 7ffb6055a640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffb4406be30 0x7ffb4406e2f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.440+0000 7ffb6055a640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb5815e930 0x7ffb581f72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.440+0000 7ffb6055a640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb5815dac0 0x7ffb5814b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.440+0000 7ffb6055a640 1 --2- 172.21.15.16:0/4037452301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb580cff10 0x7ffb580c9bc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:09.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.440+0000 7ffb6055a640 1 -- 172.21.15.16:0/4037452301 >> 172.21.15.16:0/4037452301 conn(0x7ffb580d4050 msgr2=0x7ffb580b7080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:09.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.440+0000 7ffb6055a640 1 -- 172.21.15.16:0/4037452301 shutdown_connections 2024-04-24T23:49:09.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:09.440+0000 7ffb6055a640 1 -- 172.21.15.16:0/4037452301 wait complete. 2024-04-24T23:49:09.450 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-04-24T23:49:09.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:09.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-24T23:49:09.473 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi016.stderr:ceph-fuse[41178]: fuse finished with error 0 and tester_r 0 2024-04-24T23:49:09.474 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:49:15.576 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:49:15.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:15.577 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-24T23:49:15.604 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:15.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.757+0000 7f7f51da2640 1 Processor -- start 2024-04-24T23:49:15.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.758+0000 7f7f51da2640 1 -- start start 2024-04-24T23:49:15.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.758+0000 7f7f51da2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f4c0b2bb0 0x7f7f4c0cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:15.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.758+0000 7f7f51da2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7f4c0d1fa0 0x7f7f4c0d23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:15.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.758+0000 7f7f51da2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f4c0b3390 con 0x7f7f4c15e020 2024-04-24T23:49:15.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.758+0000 7f7f51da2640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f7f4c06bfc0 con 0x7f7f4c0b2bb0 2024-04-24T23:49:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.758+0000 7f7f51da2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f7f4c06c2a0 con 0x7f7f4c0d1fa0 2024-04-24T23:49:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.759+0000 7f7f4affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f4c0b2bb0 0x7f7f4c0cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.759+0000 7f7f4affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f4c0b2bb0 0x7f7f4c0cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44332/0 (socket says 172.21.15.16:44332) 2024-04-24T23:49:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.759+0000 7f7f4affd640 1 -- 172.21.15.16:0/1861963695 learned_addr learned my addr 172.21.15.16:0/1861963695 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.759+0000 7f7f4bfff640 1 --2- 172.21.15.16:0/1861963695 >> v2:172.21.15.156:3300/0 conn(0x7f7f4c0d1fa0 0x7f7f4c0d23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.759+0000 7f7f4affd640 1 -- 172.21.15.16:0/1861963695 >> v2:172.21.15.156:3300/0 conn(0x7f7f4c0d1fa0 msgr2=0x7f7f4c0d23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.759+0000 7f7f4affd640 1 --2- 172.21.15.16:0/1861963695 >> v2:172.21.15.156:3300/0 conn(0x7f7f4c0d1fa0 0x7f7f4c0d23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.759+0000 7f7f4affd640 1 -- 172.21.15.16:0/1861963695 >> v1:172.21.15.156:6789/0 conn(0x7f7f4c15e020 legacy=0x7f7f4c0cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.759+0000 7f7f4affd640 1 -- 172.21.15.16:0/1861963695 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f4c06c580 con 0x7f7f4c0b2bb0 2024-04-24T23:49:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.760+0000 7f7f4affd640 1 --2- 172.21.15.16:0/1861963695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f4c0b2bb0 0x7f7f4c0cf980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0c78f0 tx=0x7f7f4002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18d578f56a9af297 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.760+0000 7f7f4a7fc640 1 -- 172.21.15.16:0/1861963695 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f40037070 con 0x7f7f4c0b2bb0 2024-04-24T23:49:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.760+0000 7f7f4a7fc640 1 -- 172.21.15.16:0/1861963695 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f40032440 con 0x7f7f4c0b2bb0 2024-04-24T23:49:15.761 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.760+0000 7f7f4a7fc640 1 -- 172.21.15.16:0/1861963695 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f40032810 con 0x7f7f4c0b2bb0 2024-04-24T23:49:15.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.760+0000 7f7f51da2640 1 -- 172.21.15.16:0/1861963695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f4c0b2bb0 msgr2=0x7f7f4c0cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:15.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.760+0000 7f7f51da2640 1 --2- 172.21.15.16:0/1861963695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f4c0b2bb0 0x7f7f4c0cf980 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c0c78f0 tx=0x7f7f4002a1e0 comp rx=0 tx=0).stop 2024-04-24T23:49:15.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.760+0000 7f7f51da2640 1 -- 172.21.15.16:0/1861963695 shutdown_connections 2024-04-24T23:49:15.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.760+0000 7f7f51da2640 1 --2- 172.21.15.16:0/1861963695 >> v2:172.21.15.156:3300/0 conn(0x7f7f4c0d1fa0 0x7f7f4c0d23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:15.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.760+0000 7f7f51da2640 1 --2- 172.21.15.16:0/1861963695 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f4c0b2bb0 0x7f7f4c0cf980 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:15.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.760+0000 7f7f51da2640 1 -- 172.21.15.16:0/1861963695 >> 172.21.15.16:0/1861963695 conn(0x7f7f4c0d3e10 msgr2=0x7f7f4c0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:15.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.761+0000 7f7f51da2640 1 -- 172.21.15.16:0/1861963695 shutdown_connections 2024-04-24T23:49:15.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.761+0000 7f7f51da2640 1 -- 172.21.15.16:0/1861963695 wait complete. 2024-04-24T23:49:15.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.762+0000 7f7f51da2640 1 Processor -- start 2024-04-24T23:49:15.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.762+0000 7f7f51da2640 1 -- start start 2024-04-24T23:49:15.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f51da2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f4c0b2bb0 0x7f7f4c157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:15.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f51da2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f4c0d1fa0 0x7f7f4c0cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f51da2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f4c15e020 0x7f7f4c0b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f51da2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4c15e400 con 0x7f7f4c0d1fa0 2024-04-24T23:49:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f51da2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4c0d1a80 con 0x7f7f4c0b2bb0 2024-04-24T23:49:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f51da2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7f4c0cfec0 con 0x7f7f4c15e020 2024-04-24T23:49:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f4affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f4c0d1fa0 0x7f7f4c0cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f4affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f4c0d1fa0 0x7f7f4c0cd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44344/0 (socket says 172.21.15.16:44344) 2024-04-24T23:49:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f4affd640 1 -- 172.21.15.16:0/2929749747 learned_addr learned my addr 172.21.15.16:0/2929749747 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f4bfff640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f4c15e020 0x7f7f4c0b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f4b7fe640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f4c0b2bb0 0x7f7f4c157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f4affd640 1 -- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f4c15e020 msgr2=0x7f7f4c0b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f4affd640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f4c15e020 0x7f7f4c0b3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f4affd640 1 -- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f4c0b2bb0 msgr2=0x7f7f4c157230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:15.765 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f4affd640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f4c0b2bb0 0x7f7f4c157230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f4affd640 1 -- 172.21.15.16:0/2929749747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f4c0b4330 con 0x7f7f4c0d1fa0 2024-04-24T23:49:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f4bfff640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f4c15e020 0x7f7f4c0b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f4b7fe640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f4c0b2bb0 0x7f7f4c157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f4affd640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f4c0d1fa0 0x7f7f4c0cd760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7f4002a220 tx=0x7f7f400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f33fff640 1 -- 172.21.15.16:0/2929749747 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f40037040 con 0x7f7f4c0d1fa0 2024-04-24T23:49:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f33fff640 1 -- 172.21.15.16:0/2929749747 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f40042070 con 0x7f7f4c0d1fa0 2024-04-24T23:49:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f51da2640 1 -- 172.21.15.16:0/2929749747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f4c0b45d0 con 0x7f7f4c0d1fa0 2024-04-24T23:49:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.763+0000 7f7f33fff640 1 -- 172.21.15.16:0/2929749747 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f40030880 con 0x7f7f4c0d1fa0 2024-04-24T23:49:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.764+0000 7f7f51da2640 1 -- 172.21.15.16:0/2929749747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f4c158d30 con 0x7f7f4c0d1fa0 2024-04-24T23:49:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.765+0000 7f7f51da2640 1 -- 172.21.15.16:0/2929749747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f4c0cf270 con 0x7f7f4c0d1fa0 2024-04-24T23:49:15.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.765+0000 7f7f33fff640 1 -- 172.21.15.16:0/2929749747 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7f40030a20 con 0x7f7f4c0d1fa0 2024-04-24T23:49:15.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.765+0000 7f7f33fff640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7f3806be60 0x7f7f3806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.765+0000 7f7f33fff640 1 -- 172.21.15.16:0/2929749747 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7f400ab9e0 con 0x7f7f4c0d1fa0 2024-04-24T23:49:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.766+0000 7f7f4b7fe640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7f3806be60 0x7f7f3806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.766+0000 7f7f4b7fe640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7f3806be60 0x7f7f3806e320 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c001660 tx=0x7f7f3c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:15.770 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.770+0000 7f7f33fff640 1 -- 172.21.15.16:0/2929749747 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7f4c0cf270 con 0x7f7f4c0d1fa0 2024-04-24T23:49:15.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.953+0000 7f7f51da2640 1 -- 172.21.15.16:0/2929749747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7f4c06dfe0 con 0x7f7f4c0d1fa0 2024-04-24T23:49:15.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.954+0000 7f7f33fff640 1 -- 172.21.15.16:0/2929749747 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18502 (secure 0 0 0) 0x7f7f4007a050 con 0x7f7f4c0d1fa0 2024-04-24T23:49:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:15.954 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:59.888015+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.956+0000 7f7f51da2640 1 -- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7f3806be60 msgr2=0x7f7f3806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.956+0000 7f7f51da2640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7f3806be60 0x7f7f3806e320 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c001660 tx=0x7f7f3c009040 comp rx=0 tx=0).stop 2024-04-24T23:49:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.956+0000 7f7f51da2640 1 -- 172.21.15.16:0/2929749747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f4c0d1fa0 msgr2=0x7f7f4c0cd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.956+0000 7f7f51da2640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f4c0d1fa0 0x7f7f4c0cd760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7f4002a220 tx=0x7f7f400046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.956+0000 7f7f51da2640 1 -- 172.21.15.16:0/2929749747 shutdown_connections 2024-04-24T23:49:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.956+0000 7f7f51da2640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7f3806be60 0x7f7f3806e320 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.956+0000 7f7f51da2640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f4c15e020 0x7f7f4c0b3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.956+0000 7f7f51da2640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f4c0d1fa0 0x7f7f4c0cd760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:15.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.956+0000 7f7f51da2640 1 --2- 172.21.15.16:0/2929749747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f4c0b2bb0 0x7f7f4c157230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:15.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.956+0000 7f7f51da2640 1 -- 172.21.15.16:0/2929749747 >> 172.21.15.16:0/2929749747 conn(0x7f7f4c0d3e10 msgr2=0x7f7f4c15ca60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:15.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.956+0000 7f7f51da2640 1 -- 172.21.15.16:0/2929749747 shutdown_connections 2024-04-24T23:49:15.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:15.956+0000 7f7f51da2640 1 -- 172.21.15.16:0/2929749747 wait complete. 2024-04-24T23:49:15.970 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:16.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.071+0000 7f78e95d5640 1 Processor -- start 2024-04-24T23:49:16.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.071+0000 7f78e95d5640 1 -- start start 2024-04-24T23:49:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.072+0000 7f78e95d5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e415e0c0 0x7f78e40ceba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.072+0000 7f78e95d5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78e4162730 0x7f78e40b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.072+0000 7f78e95d5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78e40d20e0 con 0x7f78e40b75f0 2024-04-24T23:49:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.072+0000 7f78e95d5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f78e40d23c0 con 0x7f78e4162730 2024-04-24T23:49:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.072+0000 7f78e95d5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f78e40d26a0 con 0x7f78e415e0c0 2024-04-24T23:49:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.072+0000 7f78e27fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78e4162730 0x7f78e40b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.072+0000 7f78e37fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f78e40b75f0 0x7f78e40d3180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49712/0 (socket says 172.21.15.16:49712) 2024-04-24T23:49:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.072+0000 7f78e37fe640 1 -- 172.21.15.16:0/1435990266 learned_addr learned my addr 172.21.15.16:0/1435990266 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.072+0000 7f78e2ffd640 1 --2- 172.21.15.16:0/1435990266 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e415e0c0 0x7f78e40ceba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.073+0000 7f78e1ffb640 1 -- 172.21.15.16:0/1435990266 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2129811771 0 0) 0x7f78e40d20e0 con 0x7f78e40b75f0 2024-04-24T23:49:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.073+0000 7f78e27fc640 1 -- 172.21.15.16:0/1435990266 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e415e0c0 msgr2=0x7f78e40ceba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.073+0000 7f78e27fc640 1 --2- 172.21.15.16:0/1435990266 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e415e0c0 0x7f78e40ceba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.073+0000 7f78e27fc640 1 -- 172.21.15.16:0/1435990266 >> v1:172.21.15.16:6789/0 conn(0x7f78e40b75f0 legacy=0x7f78e40d3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.073+0000 7f78e27fc640 1 -- 172.21.15.16:0/1435990266 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e40d2980 con 0x7f78e4162730 2024-04-24T23:49:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.073+0000 7f78e27fc640 1 --2- 172.21.15.16:0/1435990266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78e4162730 0x7f78e40b70b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0097a0 tx=0x7f78cc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68587b8870cc423c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.074+0000 7f78e1ffb640 1 -- 172.21.15.16:0/1435990266 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc037070 con 0x7f78e4162730 2024-04-24T23:49:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.074+0000 7f78e1ffb640 1 -- 172.21.15.16:0/1435990266 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78cc032440 con 0x7f78e4162730 2024-04-24T23:49:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.074+0000 7f78e1ffb640 1 -- 172.21.15.16:0/1435990266 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78cc032810 con 0x7f78e4162730 2024-04-24T23:49:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.074+0000 7f78e95d5640 1 -- 172.21.15.16:0/1435990266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78e4162730 msgr2=0x7f78e40b70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.074+0000 7f78e95d5640 1 --2- 172.21.15.16:0/1435990266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78e4162730 0x7f78e40b70b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f78cc0097a0 tx=0x7f78cc02a180 comp rx=0 tx=0).stop 2024-04-24T23:49:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.074+0000 7f78e95d5640 1 -- 172.21.15.16:0/1435990266 shutdown_connections 2024-04-24T23:49:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.074+0000 7f78e95d5640 1 --2- 172.21.15.16:0/1435990266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78e4162730 0x7f78e40b70b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.074+0000 7f78e95d5640 1 --2- 172.21.15.16:0/1435990266 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e415e0c0 0x7f78e40ceba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.074+0000 7f78e95d5640 1 -- 172.21.15.16:0/1435990266 >> 172.21.15.16:0/1435990266 conn(0x7f78e40d41f0 msgr2=0x7f78e40d4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.074+0000 7f78e95d5640 1 -- 172.21.15.16:0/1435990266 shutdown_connections 2024-04-24T23:49:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.074+0000 7f78e95d5640 1 -- 172.21.15.16:0/1435990266 wait complete. 2024-04-24T23:49:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.075+0000 7f78e95d5640 1 Processor -- start 2024-04-24T23:49:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.075+0000 7f78e95d5640 1 -- start start 2024-04-24T23:49:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.076+0000 7f78e95d5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e40b75f0 0x7f78e41612c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.076+0000 7f78e95d5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78e415e0c0 0x7f78e40c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.076+0000 7f78e95d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78e4162730 0x7f78e4145a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.076+0000 7f78e95d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f78e415e4a0 con 0x7f78e4162730 2024-04-24T23:49:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.076+0000 7f78e95d5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f78e4162b90 con 0x7f78e415e0c0 2024-04-24T23:49:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.076+0000 7f78e95d5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f78e415e620 con 0x7f78e40b75f0 2024-04-24T23:49:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.076+0000 7f78e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78e4162730 0x7f78e4145a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.076+0000 7f78e37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78e4162730 0x7f78e4145a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44360/0 (socket says 172.21.15.16:44360) 2024-04-24T23:49:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.076+0000 7f78e37fe640 1 -- 172.21.15.16:0/640385916 learned_addr learned my addr 172.21.15.16:0/640385916 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.077+0000 7f78e2ffd640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e40b75f0 0x7f78e41612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.077+0000 7f78e27fc640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78e415e0c0 0x7f78e40c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.077+0000 7f78e37fe640 1 -- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e40b75f0 msgr2=0x7f78e41612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.077+0000 7f78e37fe640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e40b75f0 0x7f78e41612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.077+0000 7f78e37fe640 1 -- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78e415e0c0 msgr2=0x7f78e40c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:16.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.077+0000 7f78e37fe640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78e415e0c0 0x7f78e40c99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.077+0000 7f78e37fe640 1 -- 172.21.15.16:0/640385916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e40c9ee0 con 0x7f78e4162730 2024-04-24T23:49:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.077+0000 7f78e2ffd640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e40b75f0 0x7f78e41612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.077+0000 7f78e37fe640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78e4162730 0x7f78e4145a10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f78dc00c450 tx=0x7f78dc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.077+0000 7f78db7fe640 1 -- 172.21.15.16:0/640385916 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc018070 con 0x7f78e4162730 2024-04-24T23:49:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.077+0000 7f78db7fe640 1 -- 172.21.15.16:0/640385916 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78dc007500 con 0x7f78e4162730 2024-04-24T23:49:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.077+0000 7f78e95d5640 1 -- 172.21.15.16:0/640385916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78e40cb730 con 0x7f78e4162730 2024-04-24T23:49:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.077+0000 7f78db7fe640 1 -- 172.21.15.16:0/640385916 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78dc004a20 con 0x7f78e4162730 2024-04-24T23:49:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.077+0000 7f78e95d5640 1 -- 172.21.15.16:0/640385916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78e4146840 con 0x7f78e4162730 2024-04-24T23:49:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.078+0000 7f78e95d5640 1 -- 172.21.15.16:0/640385916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78e40730a0 con 0x7f78e4162730 2024-04-24T23:49:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.078+0000 7f78db7fe640 1 -- 172.21.15.16:0/640385916 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f78dc01b070 con 0x7f78e4162730 2024-04-24T23:49:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.079+0000 7f78db7fe640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f78c406be30 0x7f78c406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.079+0000 7f78db7fe640 1 -- 172.21.15.16:0/640385916 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f78dc08a400 con 0x7f78e4162730 2024-04-24T23:49:16.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.079+0000 7f78e2ffd640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f78c406be30 0x7f78c406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:16.081 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.080+0000 7f78e2ffd640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f78c406be30 0x7f78c406e2f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f78e40b9cc0 tx=0x7f78d400b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:16.083 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.083+0000 7f78db7fe640 1 -- 172.21.15.16:0/640385916 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f78dc058a70 con 0x7f78e4162730 2024-04-24T23:49:16.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.267+0000 7f78e95d5640 1 -- 172.21.15.16:0/640385916 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f78e406b180 con 0x7f78e4162730 2024-04-24T23:49:16.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.269+0000 7f78db7fe640 1 -- 172.21.15.16:0/640385916 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18502 (secure 0 0 0) 0x7f78dc058370 con 0x7f78e4162730 2024-04-24T23:49:16.269 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:16.269 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:59.888015+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.271+0000 7f78e95d5640 1 -- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f78c406be30 msgr2=0x7f78c406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.271+0000 7f78e95d5640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f78c406be30 0x7f78c406e2f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f78e40b9cc0 tx=0x7f78d400b040 comp rx=0 tx=0).stop 2024-04-24T23:49:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.271+0000 7f78e95d5640 1 -- 172.21.15.16:0/640385916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78e4162730 msgr2=0x7f78e4145a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:16.271 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.271+0000 7f78e95d5640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78e4162730 0x7f78e4145a10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f78dc00c450 tx=0x7f78dc00c910 comp rx=0 tx=0).stop 2024-04-24T23:49:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.271+0000 7f78e95d5640 1 -- 172.21.15.16:0/640385916 shutdown_connections 2024-04-24T23:49:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.271+0000 7f78e95d5640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f78c406be30 0x7f78c406e2f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.271+0000 7f78e95d5640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78e4162730 0x7f78e4145a10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.271+0000 7f78e95d5640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78e415e0c0 0x7f78e40c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.271+0000 7f78e95d5640 1 --2- 172.21.15.16:0/640385916 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78e40b75f0 0x7f78e41612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.271+0000 7f78e95d5640 1 -- 172.21.15.16:0/640385916 >> 172.21.15.16:0/640385916 conn(0x7f78e40d41f0 msgr2=0x7f78e40d0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.271+0000 7f78e95d5640 1 -- 172.21.15.16:0/640385916 shutdown_connections 2024-04-24T23:49:16.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.271+0000 7f78e95d5640 1 -- 172.21.15.16:0/640385916 wait complete. 2024-04-24T23:49:16.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-24T23:49:16.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.391+0000 7fe4b51a9640 1 Processor -- start 2024-04-24T23:49:16.391 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.391+0000 7fe4b51a9640 1 -- start start 2024-04-24T23:49:16.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4b51a9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4b01478d0 0x7fe4b00d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:16.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4b51a9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe4b00d2090 0x7fe4b00b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:16.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4b51a9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4b00b2a10 con 0x7fe4b00b21d0 2024-04-24T23:49:16.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4b51a9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fe4b00d24f0 con 0x7fe4b00d2090 2024-04-24T23:49:16.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4b51a9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe4b00b3620 con 0x7fe4b01478d0 2024-04-24T23:49:16.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4af59e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe4b00b21d0 0x7fe4b00d0340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49726/0 (socket says 172.21.15.16:49726) 2024-04-24T23:49:16.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4af59e640 1 -- 172.21.15.16:0/2422467838 learned_addr learned my addr 172.21.15.16:0/2422467838 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:16.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4a7fff640 1 --2- 172.21.15.16:0/2422467838 >> v2:172.21.15.156:3300/0 conn(0x7fe4b00d2090 0x7fe4b00b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:16.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4aed9d640 1 --2- 172.21.15.16:0/2422467838 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4b01478d0 0x7fe4b00d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:16.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4ae59c640 1 -- 172.21.15.16:0/2422467838 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3561597547 0 0) 0x7fe4b00b2a10 con 0x7fe4b00b21d0 2024-04-24T23:49:16.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4ae59c640 1 -- 172.21.15.16:0/2422467838 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe49c003520 con 0x7fe4b00b21d0 2024-04-24T23:49:16.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4aed9d640 1 -- 172.21.15.16:0/2422467838 >> v2:172.21.15.156:3300/0 conn(0x7fe4b00d2090 msgr2=0x7fe4b00b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:16.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4aed9d640 1 --2- 172.21.15.16:0/2422467838 >> v2:172.21.15.156:3300/0 conn(0x7fe4b00d2090 0x7fe4b00b1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:16.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4aed9d640 1 -- 172.21.15.16:0/2422467838 >> v1:172.21.15.16:6789/0 conn(0x7fe4b00b21d0 legacy=0x7fe4b00d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:16.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4aed9d640 1 -- 172.21.15.16:0/2422467838 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b00b2cf0 con 0x7fe4b01478d0 2024-04-24T23:49:16.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.392+0000 7fe4a7fff640 1 --2- 172.21.15.16:0/2422467838 >> v2:172.21.15.156:3300/0 conn(0x7fe4b00d2090 0x7fe4b00b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:16.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.393+0000 7fe4aed9d640 1 --2- 172.21.15.16:0/2422467838 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4b01478d0 0x7fe4b00d1b50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00097a0 tx=0x7fe4a002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4dbbd15f94a5530 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:16.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.393+0000 7fe4ae59c640 1 -- 172.21.15.16:0/2422467838 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0037070 con 0x7fe4b01478d0 2024-04-24T23:49:16.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.393+0000 7fe4ae59c640 1 -- 172.21.15.16:0/2422467838 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4a0032440 con 0x7fe4b01478d0 2024-04-24T23:49:16.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.393+0000 7fe4ae59c640 1 -- 172.21.15.16:0/2422467838 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a00327f0 con 0x7fe4b01478d0 2024-04-24T23:49:16.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.394+0000 7fe4b51a9640 1 -- 172.21.15.16:0/2422467838 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4b01478d0 msgr2=0x7fe4b00d1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:16.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.394+0000 7fe4b51a9640 1 --2- 172.21.15.16:0/2422467838 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4b01478d0 0x7fe4b00d1b50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00097a0 tx=0x7fe4a002a180 comp rx=0 tx=0).stop 2024-04-24T23:49:16.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.394+0000 7fe4b51a9640 1 -- 172.21.15.16:0/2422467838 shutdown_connections 2024-04-24T23:49:16.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.394+0000 7fe4b51a9640 1 --2- 172.21.15.16:0/2422467838 >> v2:172.21.15.156:3300/0 conn(0x7fe4b00d2090 0x7fe4b00b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:16.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.394+0000 7fe4b51a9640 1 --2- 172.21.15.16:0/2422467838 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4b01478d0 0x7fe4b00d1b50 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:16.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.394+0000 7fe4b51a9640 1 -- 172.21.15.16:0/2422467838 >> 172.21.15.16:0/2422467838 conn(0x7fe4b006b2c0 msgr2=0x7fe4b0072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:16.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.394+0000 7fe4b51a9640 1 -- 172.21.15.16:0/2422467838 shutdown_connections 2024-04-24T23:49:16.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.394+0000 7fe4b51a9640 1 -- 172.21.15.16:0/2422467838 wait complete. 2024-04-24T23:49:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.395+0000 7fe4b51a9640 1 Processor -- start 2024-04-24T23:49:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.395+0000 7fe4b51a9640 1 -- start start 2024-04-24T23:49:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4b51a9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4b00b21d0 0x7fe4b013aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4b51a9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4b00d2090 0x7fe4b01407f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4b51a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4b01478d0 0x7fe4b0140d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4b51a9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b00b2660 con 0x7fe4b01478d0 2024-04-24T23:49:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4b51a9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe4b00b32b0 con 0x7fe4b00b21d0 2024-04-24T23:49:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4b51a9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe4b015e950 con 0x7fe4b00d2090 2024-04-24T23:49:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4af59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4b01478d0 0x7fe4b0140d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4af59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4b01478d0 0x7fe4b0140d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:44362/0 (socket says 172.21.15.16:44362) 2024-04-24T23:49:16.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4af59e640 1 -- 172.21.15.16:0/3662455252 learned_addr learned my addr 172.21.15.16:0/3662455252 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4aed9d640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4b00b21d0 0x7fe4b013aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4a7fff640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4b00d2090 0x7fe4b01407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4af59e640 1 -- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4b00d2090 msgr2=0x7fe4b01407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4af59e640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4b00d2090 0x7fe4b01407f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4af59e640 1 -- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4b00b21d0 msgr2=0x7fe4b013aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4af59e640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4b00b21d0 0x7fe4b013aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4af59e640 1 -- 172.21.15.16:0/3662455252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4b015e030 con 0x7fe4b01478d0 2024-04-24T23:49:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4a7fff640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4b00d2090 0x7fe4b01407f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.396+0000 7fe4af59e640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4b01478d0 0x7fe4b0140d30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe4a800caa0 tx=0x7fe4a800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.397+0000 7fe4a77fe640 1 -- 172.21.15.16:0/3662455252 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a8017070 con 0x7fe4b01478d0 2024-04-24T23:49:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.397+0000 7fe4a77fe640 1 -- 172.21.15.16:0/3662455252 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4a800ac00 con 0x7fe4b01478d0 2024-04-24T23:49:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.397+0000 7fe4a77fe640 1 -- 172.21.15.16:0/3662455252 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4a8012400 con 0x7fe4b01478d0 2024-04-24T23:49:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.397+0000 7fe4b51a9640 1 -- 172.21.15.16:0/3662455252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4b0141980 con 0x7fe4b01478d0 2024-04-24T23:49:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.397+0000 7fe4b51a9640 1 -- 172.21.15.16:0/3662455252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4b01416a0 con 0x7fe4b01478d0 2024-04-24T23:49:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.398+0000 7fe4b51a9640 1 -- 172.21.15.16:0/3662455252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4b0068f90 con 0x7fe4b01478d0 2024-04-24T23:49:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.398+0000 7fe4a77fe640 1 -- 172.21.15.16:0/3662455252 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe4a801a070 con 0x7fe4b01478d0 2024-04-24T23:49:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.399+0000 7fe4a77fe640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe49006be30 0x7fe49006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.399+0000 7fe4a77fe640 1 -- 172.21.15.16:0/3662455252 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe4a808af20 con 0x7fe4b01478d0 2024-04-24T23:49:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.399+0000 7fe4aed9d640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe49006be30 0x7fe49006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.399+0000 7fe4aed9d640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe49006be30 0x7fe49006e2f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0034000 tx=0x7fe4a002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:16.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.403+0000 7fe4a77fe640 1 -- 172.21.15.16:0/3662455252 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe4a8059580 con 0x7fe4b01478d0 2024-04-24T23:49:16.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:16.626+0000 7fe4b51a9640 1 -- 172.21.15.16:0/3662455252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fe4b006fd90 con 0x7fe4b01478d0 2024-04-24T23:49:17.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.550+0000 7fe4a77fe640 1 -- 172.21.15.16:0/3662455252 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) ==== 118+0+0 (secure 0 0 0) 0x7fe4a8058e80 con 0x7fe4b01478d0 2024-04-24T23:49:17.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.553+0000 7fe4b51a9640 1 -- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe49006be30 msgr2=0x7fe49006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:17.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.553+0000 7fe4b51a9640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe49006be30 0x7fe49006e2f0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0034000 tx=0x7fe4a002e000 comp rx=0 tx=0).stop 2024-04-24T23:49:17.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.553+0000 7fe4b51a9640 1 -- 172.21.15.16:0/3662455252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4b01478d0 msgr2=0x7fe4b0140d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:17.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.553+0000 7fe4b51a9640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4b01478d0 0x7fe4b0140d30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe4a800caa0 tx=0x7fe4a800cf60 comp rx=0 tx=0).stop 2024-04-24T23:49:17.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.554+0000 7fe4b51a9640 1 -- 172.21.15.16:0/3662455252 shutdown_connections 2024-04-24T23:49:17.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.554+0000 7fe4b51a9640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe49006be30 0x7fe49006e2f0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:17.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.554+0000 7fe4b51a9640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4b01478d0 0x7fe4b0140d30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:17.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.554+0000 7fe4b51a9640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe4b00d2090 0x7fe4b01407f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:17.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.554+0000 7fe4b51a9640 1 --2- 172.21.15.16:0/3662455252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe4b00b21d0 0x7fe4b013aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:17.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.554+0000 7fe4b51a9640 1 -- 172.21.15.16:0/3662455252 >> 172.21.15.16:0/3662455252 conn(0x7fe4b006b2c0 msgr2=0x7fe4b00b56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:17.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.554+0000 7fe4b51a9640 1 -- 172.21.15.16:0/3662455252 shutdown_connections 2024-04-24T23:49:17.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.554+0000 7fe4b51a9640 1 -- 172.21.15.16:0/3662455252 wait complete. 2024-04-24T23:49:17.567 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-04-24T23:49:17.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-24T23:49:17.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.678+0000 7fae7e07d640 1 Processor -- start 2024-04-24T23:49:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.678+0000 7fae7e07d640 1 -- start start 2024-04-24T23:49:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.678+0000 7fae7e07d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae7806bc40 0x7fae7806c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.678+0000 7fae7e07d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fae780b7050 0x7fae780b7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.678+0000 7fae7e07d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae780d24d0 con 0x7fae7815dfd0 2024-04-24T23:49:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.679+0000 7fae7e07d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fae7806ba00 con 0x7fae7806bc40 2024-04-24T23:49:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.679+0000 7fae7e07d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fae780d27b0 con 0x7fae780b7050 2024-04-24T23:49:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.679+0000 7fae76ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae7806bc40 0x7fae7806c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.679+0000 7fae76ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae7806bc40 0x7fae7806c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48228/0 (socket says 172.21.15.16:48228) 2024-04-24T23:49:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.679+0000 7fae76ffd640 1 -- 172.21.15.16:0/2233177620 learned_addr learned my addr 172.21.15.16:0/2233177620 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.679+0000 7fae77fff640 1 --2- 172.21.15.16:0/2233177620 >> v2:172.21.15.156:3300/0 conn(0x7fae780b7050 0x7fae780b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.679+0000 7fae76ffd640 1 -- 172.21.15.16:0/2233177620 >> v2:172.21.15.156:3300/0 conn(0x7fae780b7050 msgr2=0x7fae780b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.679+0000 7fae76ffd640 1 --2- 172.21.15.16:0/2233177620 >> v2:172.21.15.156:3300/0 conn(0x7fae780b7050 0x7fae780b7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.679+0000 7fae76ffd640 1 -- 172.21.15.16:0/2233177620 >> v1:172.21.15.156:6789/0 conn(0x7fae7815dfd0 legacy=0x7fae7806b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.679+0000 7fae76ffd640 1 -- 172.21.15.16:0/2233177620 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae780d2a90 con 0x7fae7806bc40 2024-04-24T23:49:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.680+0000 7fae76ffd640 1 --2- 172.21.15.16:0/2233177620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae7806bc40 0x7fae7806c0d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fae780b6ca0 tx=0x7fae6c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac8b01f98233fccc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.680+0000 7fae767fc640 1 -- 172.21.15.16:0/2233177620 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c037070 con 0x7fae7806bc40 2024-04-24T23:49:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.680+0000 7fae767fc640 1 -- 172.21.15.16:0/2233177620 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae6c032440 con 0x7fae7806bc40 2024-04-24T23:49:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.680+0000 7fae767fc640 1 -- 172.21.15.16:0/2233177620 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c032810 con 0x7fae7806bc40 2024-04-24T23:49:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.680+0000 7fae7e07d640 1 -- 172.21.15.16:0/2233177620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae7806bc40 msgr2=0x7fae7806c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.680+0000 7fae7e07d640 1 --2- 172.21.15.16:0/2233177620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae7806bc40 0x7fae7806c0d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fae780b6ca0 tx=0x7fae6c02a1a0 comp rx=0 tx=0).stop 2024-04-24T23:49:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.680+0000 7fae7e07d640 1 -- 172.21.15.16:0/2233177620 shutdown_connections 2024-04-24T23:49:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.680+0000 7fae7e07d640 1 --2- 172.21.15.16:0/2233177620 >> v2:172.21.15.156:3300/0 conn(0x7fae780b7050 0x7fae780b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.680+0000 7fae7e07d640 1 --2- 172.21.15.16:0/2233177620 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae7806bc40 0x7fae7806c0d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.680+0000 7fae7e07d640 1 -- 172.21.15.16:0/2233177620 >> 172.21.15.16:0/2233177620 conn(0x7fae780d3dc0 msgr2=0x7fae780d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.680+0000 7fae7e07d640 1 -- 172.21.15.16:0/2233177620 shutdown_connections 2024-04-24T23:49:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.680+0000 7fae7e07d640 1 -- 172.21.15.16:0/2233177620 wait complete. 2024-04-24T23:49:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.682+0000 7fae7e07d640 1 Processor -- start 2024-04-24T23:49:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.682+0000 7fae7e07d640 1 -- start start 2024-04-24T23:49:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.682+0000 7fae7e07d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae7806bc40 0x7fae78158670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.682+0000 7fae7e07d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae780b7050 0x7fae78157fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.682+0000 7fae7e07d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae7815dfd0 0x7fae781f4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.682+0000 7fae7e07d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fae780cead0 con 0x7fae7806bc40 2024-04-24T23:49:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.682+0000 7fae7e07d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fae780d3050 con 0x7fae7815dfd0 2024-04-24T23:49:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.682+0000 7fae7e07d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fae780b79a0 con 0x7fae780b7050 2024-04-24T23:49:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae7806bc40 0x7fae78158670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:17.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae777fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae7806bc40 0x7fae78158670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48232/0 (socket says 172.21.15.16:48232) 2024-04-24T23:49:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae777fe640 1 -- 172.21.15.16:0/1647504217 learned_addr learned my addr 172.21.15.16:0/1647504217 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae77fff640 1 --2- 172.21.15.16:0/1647504217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae7815dfd0 0x7fae781f4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae76ffd640 1 --2- 172.21.15.16:0/1647504217 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae780b7050 0x7fae78157fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae777fe640 1 -- 172.21.15.16:0/1647504217 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae780b7050 msgr2=0x7fae78157fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae777fe640 1 --2- 172.21.15.16:0/1647504217 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae780b7050 0x7fae78157fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae777fe640 1 -- 172.21.15.16:0/1647504217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae7815dfd0 msgr2=0x7fae781f4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae777fe640 1 --2- 172.21.15.16:0/1647504217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae7815dfd0 0x7fae781f4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae777fe640 1 -- 172.21.15.16:0/1647504217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae780cd610 con 0x7fae7806bc40 2024-04-24T23:49:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae777fe640 1 --2- 172.21.15.16:0/1647504217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae7806bc40 0x7fae78158670 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fae6800b3e0 tx=0x7fae6800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae5ffff640 1 -- 172.21.15.16:0/1647504217 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae68015020 con 0x7fae7806bc40 2024-04-24T23:49:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae5ffff640 1 -- 172.21.15.16:0/1647504217 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae68011850 con 0x7fae7806bc40 2024-04-24T23:49:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae5ffff640 1 -- 172.21.15.16:0/1647504217 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae68011c40 con 0x7fae7806bc40 2024-04-24T23:49:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae7e07d640 1 -- 172.21.15.16:0/1647504217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae780cb0e0 con 0x7fae7806bc40 2024-04-24T23:49:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.683+0000 7fae7e07d640 1 -- 172.21.15.16:0/1647504217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae781f4690 con 0x7fae7806bc40 2024-04-24T23:49:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.684+0000 7fae7e07d640 1 -- 172.21.15.16:0/1647504217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae78068f90 con 0x7fae7806bc40 2024-04-24T23:49:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.685+0000 7fae5ffff640 1 -- 172.21.15.16:0/1647504217 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fae68018020 con 0x7fae7806bc40 2024-04-24T23:49:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.685+0000 7fae5ffff640 1 --2- 172.21.15.16:0/1647504217 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fae6406be30 0x7fae6406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.685+0000 7fae5ffff640 1 -- 172.21.15.16:0/1647504217 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fae6808b010 con 0x7fae7806bc40 2024-04-24T23:49:17.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.686+0000 7fae76ffd640 1 --2- 172.21.15.16:0/1647504217 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fae6406be30 0x7fae6406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:17.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.686+0000 7fae76ffd640 1 --2- 172.21.15.16:0/1647504217 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fae6406be30 0x7fae6406e2f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fae6c034000 tx=0x7fae6c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:17.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.690+0000 7fae5ffff640 1 -- 172.21.15.16:0/1647504217 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fae6801f020 con 0x7fae7806bc40 2024-04-24T23:49:17.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:17.913+0000 7fae7e07d640 1 -- 172.21.15.16:0/1647504217 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fae780d0340 con 0x7fae7806bc40 2024-04-24T23:49:18.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.552+0000 7fae5ffff640 1 -- 172.21.15.16:0/1647504217 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) ==== 122+0+0 (secure 0 0 0) 0x7fae68059030 con 0x7fae7806bc40 2024-04-24T23:49:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.555+0000 7fae7e07d640 1 -- 172.21.15.16:0/1647504217 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fae6406be30 msgr2=0x7fae6406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.555+0000 7fae7e07d640 1 --2- 172.21.15.16:0/1647504217 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fae6406be30 0x7fae6406e2f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fae6c034000 tx=0x7fae6c02e000 comp rx=0 tx=0).stop 2024-04-24T23:49:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.555+0000 7fae7e07d640 1 -- 172.21.15.16:0/1647504217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae7806bc40 msgr2=0x7fae78158670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.555+0000 7fae7e07d640 1 --2- 172.21.15.16:0/1647504217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae7806bc40 0x7fae78158670 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fae6800b3e0 tx=0x7fae6800b8a0 comp rx=0 tx=0).stop 2024-04-24T23:49:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.555+0000 7fae7e07d640 1 -- 172.21.15.16:0/1647504217 shutdown_connections 2024-04-24T23:49:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.555+0000 7fae7e07d640 1 --2- 172.21.15.16:0/1647504217 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fae6406be30 0x7fae6406e2f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.555+0000 7fae7e07d640 1 --2- 172.21.15.16:0/1647504217 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fae7815dfd0 0x7fae781f4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:18.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.555+0000 7fae7e07d640 1 --2- 172.21.15.16:0/1647504217 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fae780b7050 0x7fae78157fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.555+0000 7fae7e07d640 1 --2- 172.21.15.16:0/1647504217 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae7806bc40 0x7fae78158670 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.555+0000 7fae7e07d640 1 -- 172.21.15.16:0/1647504217 >> 172.21.15.16:0/1647504217 conn(0x7fae780d3dc0 msgr2=0x7fae780b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.555+0000 7fae7e07d640 1 -- 172.21.15.16:0/1647504217 shutdown_connections 2024-04-24T23:49:18.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.555+0000 7fae7e07d640 1 -- 172.21.15.16:0/1647504217 wait complete. 2024-04-24T23:49:18.568 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-24T23:49:18.569 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:49:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.677+0000 7f46529ce640 1 Processor -- start 2024-04-24T23:49:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.677+0000 7f46529ce640 1 -- start start 2024-04-24T23:49:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.677+0000 7f46529ce640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f464c0cea70 0x7f464c0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.677+0000 7f46529ce640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f464c0d05a0 con 0x7f464c0ce5f0 2024-04-24T23:49:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.677+0000 7f46529ce640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f464c0d0880 con 0x7f464c15dfa0 2024-04-24T23:49:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.677+0000 7f46529ce640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f464c0b7360 con 0x7f464c0cea70 2024-04-24T23:49:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.677+0000 7f464b7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f464c0ce5f0 0x7f464c0ccad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50574/0 (socket says 172.21.15.16:50574) 2024-04-24T23:49:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.678+0000 7f464b7fe640 1 -- 172.21.15.16:0/2792281482 learned_addr learned my addr 172.21.15.16:0/2792281482 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.678+0000 7f4650f6b640 1 --2- 172.21.15.16:0/2792281482 >> v2:172.21.15.156:3300/0 conn(0x7f464c0cea70 0x7f464c0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.678+0000 7f464affd640 1 -- 172.21.15.16:0/2792281482 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1776675247 0 0) 0x7f464c0d05a0 con 0x7f464c0ce5f0 2024-04-24T23:49:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.678+0000 7f464affd640 1 -- 172.21.15.16:0/2792281482 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4634003520 con 0x7f464c0ce5f0 2024-04-24T23:49:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.678+0000 7f464affd640 1 -- 172.21.15.16:0/2792281482 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2330379519 0 0) 0x7f464c0d0880 con 0x7f464c15dfa0 2024-04-24T23:49:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.678+0000 7f464affd640 1 -- 172.21.15.16:0/2792281482 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f464c0d05a0 con 0x7f464c15dfa0 2024-04-24T23:49:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.678+0000 7f4650f6b640 1 -- 172.21.15.16:0/2792281482 >> v1:172.21.15.156:6789/0 conn(0x7f464c15dfa0 legacy=0x7f464c0d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.678+0000 7f464affd640 1 -- 172.21.15.16:0/2792281482 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2866011627 0 0) 0x7f4634003520 con 0x7f464c0ce5f0 2024-04-24T23:49:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.678+0000 7f4650f6b640 1 -- 172.21.15.16:0/2792281482 >> v1:172.21.15.16:6789/0 conn(0x7f464c0ce5f0 legacy=0x7f464c0ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.678+0000 7f4650f6b640 1 -- 172.21.15.16:0/2792281482 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f464c0d0b60 con 0x7f464c0cea70 2024-04-24T23:49:18.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.679+0000 7f4650f6b640 1 --2- 172.21.15.16:0/2792281482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f464c0cea70 0x7f464c0cd1e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4644009d90 tx=0x7f464402a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e072fdc7b925dc0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:18.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.679+0000 7f464affd640 1 -- 172.21.15.16:0/2792281482 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4640001670 con 0x7f464c0cea70 2024-04-24T23:49:18.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.679+0000 7f464affd640 1 -- 172.21.15.16:0/2792281482 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4644007540 con 0x7f464c0cea70 2024-04-24T23:49:18.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.679+0000 7f464affd640 1 -- 172.21.15.16:0/2792281482 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644007910 con 0x7f464c0cea70 2024-04-24T23:49:18.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.680+0000 7f46529ce640 1 -- 172.21.15.16:0/2792281482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f464c0cea70 msgr2=0x7f464c0cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:18.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.680+0000 7f46529ce640 1 --2- 172.21.15.16:0/2792281482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f464c0cea70 0x7f464c0cd1e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4644009d90 tx=0x7f464402a680 comp rx=0 tx=0).stop 2024-04-24T23:49:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.680+0000 7f46529ce640 1 -- 172.21.15.16:0/2792281482 shutdown_connections 2024-04-24T23:49:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.680+0000 7f46529ce640 1 --2- 172.21.15.16:0/2792281482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f464c0cea70 0x7f464c0cd1e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.680+0000 7f46529ce640 1 -- 172.21.15.16:0/2792281482 >> 172.21.15.16:0/2792281482 conn(0x7f464c06b4d0 msgr2=0x7f464c0b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.680+0000 7f46529ce640 1 -- 172.21.15.16:0/2792281482 shutdown_connections 2024-04-24T23:49:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.680+0000 7f46529ce640 1 -- 172.21.15.16:0/2792281482 wait complete. 2024-04-24T23:49:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.681+0000 7f46529ce640 1 Processor -- start 2024-04-24T23:49:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.681+0000 7f46529ce640 1 -- start start 2024-04-24T23:49:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f46529ce640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f464c0ce5f0 0x7f464c14e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f46529ce640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f464c0cea70 0x7f464c152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f46529ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f464c15dfa0 0x7f464c0cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f46529ce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f464c0ceee0 con 0x7f464c15dfa0 2024-04-24T23:49:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f46529ce640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f464c0cc640 con 0x7f464c0ce5f0 2024-04-24T23:49:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f46529ce640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f464c15e380 con 0x7f464c0cea70 2024-04-24T23:49:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f4650f6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f464c15dfa0 0x7f464c0cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f4650f6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f464c15dfa0 0x7f464c0cafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48240/0 (socket says 172.21.15.16:48240) 2024-04-24T23:49:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f4650f6b640 1 -- 172.21.15.16:0/3689423980 learned_addr learned my addr 172.21.15.16:0/3689423980 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f464bfff640 1 --2- 172.21.15.16:0/3689423980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f464c0ce5f0 0x7f464c14e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f464b7fe640 1 --2- 172.21.15.16:0/3689423980 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f464c0cea70 0x7f464c152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f4650f6b640 1 -- 172.21.15.16:0/3689423980 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f464c0cea70 msgr2=0x7f464c152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f4650f6b640 1 --2- 172.21.15.16:0/3689423980 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f464c0cea70 0x7f464c152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f4650f6b640 1 -- 172.21.15.16:0/3689423980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f464c0ce5f0 msgr2=0x7f464c14e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f4650f6b640 1 --2- 172.21.15.16:0/3689423980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f464c0ce5f0 0x7f464c14e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.682+0000 7f4650f6b640 1 -- 172.21.15.16:0/3689423980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f464c0c8f00 con 0x7f464c15dfa0 2024-04-24T23:49:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.683+0000 7f4650f6b640 1 --2- 172.21.15.16:0/3689423980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f464c15dfa0 0x7f464c0cafe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4644009d90 tx=0x7f4644004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.683+0000 7f4648ff9640 1 -- 172.21.15.16:0/3689423980 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644040070 con 0x7f464c15dfa0 2024-04-24T23:49:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.683+0000 7f4648ff9640 1 -- 172.21.15.16:0/3689423980 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4644031bc0 con 0x7f464c15dfa0 2024-04-24T23:49:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.683+0000 7f4648ff9640 1 -- 172.21.15.16:0/3689423980 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4644007e90 con 0x7f464c15dfa0 2024-04-24T23:49:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.683+0000 7f46529ce640 1 -- 172.21.15.16:0/3689423980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f464c0c91e0 con 0x7f464c15dfa0 2024-04-24T23:49:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.683+0000 7f46529ce640 1 -- 172.21.15.16:0/3689423980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f464c14e910 con 0x7f464c15dfa0 2024-04-24T23:49:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.684+0000 7f46529ce640 1 -- 172.21.15.16:0/3689423980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f464c06bea0 con 0x7f464c15dfa0 2024-04-24T23:49:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.684+0000 7f4648ff9640 1 -- 172.21.15.16:0/3689423980 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4644039040 con 0x7f464c15dfa0 2024-04-24T23:49:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.685+0000 7f4648ff9640 1 --2- 172.21.15.16:0/3689423980 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f463806be60 0x7f463806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.685+0000 7f4648ff9640 1 -- 172.21.15.16:0/3689423980 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f46440ac4d0 con 0x7f464c15dfa0 2024-04-24T23:49:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.685+0000 7f464bfff640 1 --2- 172.21.15.16:0/3689423980 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f463806be60 0x7f463806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.685+0000 7f464bfff640 1 --2- 172.21.15.16:0/3689423980 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f463806be60 0x7f463806e320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f463c0011e0 tx=0x7f463c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.689+0000 7f4648ff9640 1 -- 172.21.15.16:0/3689423980 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f464407a450 con 0x7f464c15dfa0 2024-04-24T23:49:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.909+0000 7f46529ce640 1 -- 172.21.15.16:0/3689423980 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f464c04f5e0 con 0x7f464c15dfa0 2024-04-24T23:49:18.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.910+0000 7f4648ff9640 1 -- 172.21.15.16:0/3689423980 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5180 (secure 0 0 0) 0x7f464407e750 con 0x7f464c15dfa0 2024-04-24T23:49:18.910 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:18.910 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1623032341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1623032341},{"type":"v1","addr":"172.21.15.156:6836","nonce":1623032341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3530738236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3530738236},{"type":"v1","addr":"172.21.15.16:6835","nonce":3530738236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2153063131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2153063131},{"type":"v1","addr":"172.21.15.16:6834","nonce":2153063131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:48:54.852787+0000","modified":"2024-04-24T23:49:17.914730+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/289127714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":289127714},{"type":"v1","addr":"172.21.15.156:6837","nonce":289127714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-24T23:49:18.910 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-04-24T23:49:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.912+0000 7f46529ce640 1 -- 172.21.15.16:0/3689423980 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f463806be60 msgr2=0x7f463806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.912+0000 7f46529ce640 1 --2- 172.21.15.16:0/3689423980 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f463806be60 0x7f463806e320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f463c0011e0 tx=0x7f463c009040 comp rx=0 tx=0).stop 2024-04-24T23:49:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.912+0000 7f46529ce640 1 -- 172.21.15.16:0/3689423980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f464c15dfa0 msgr2=0x7f464c0cafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.912+0000 7f46529ce640 1 --2- 172.21.15.16:0/3689423980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f464c15dfa0 0x7f464c0cafe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4644009d90 tx=0x7f4644004a40 comp rx=0 tx=0).stop 2024-04-24T23:49:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.912+0000 7f46529ce640 1 -- 172.21.15.16:0/3689423980 shutdown_connections 2024-04-24T23:49:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.912+0000 7f46529ce640 1 --2- 172.21.15.16:0/3689423980 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f463806be60 0x7f463806e320 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.912+0000 7f46529ce640 1 --2- 172.21.15.16:0/3689423980 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f464c15dfa0 0x7f464c0cafe0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.912+0000 7f46529ce640 1 --2- 172.21.15.16:0/3689423980 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f464c0cea70 0x7f464c152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.912+0000 7f46529ce640 1 --2- 172.21.15.16:0/3689423980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f464c0ce5f0 0x7f464c14e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.912+0000 7f46529ce640 1 -- 172.21.15.16:0/3689423980 >> 172.21.15.16:0/3689423980 conn(0x7f464c06b4d0 msgr2=0x7f464c0c84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.912+0000 7f46529ce640 1 -- 172.21.15.16:0/3689423980 shutdown_connections 2024-04-24T23:49:18.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:18.912+0000 7f46529ce640 1 -- 172.21.15.16:0/3689423980 wait complete. 2024-04-24T23:49:18.923 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:19.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.023+0000 7f6c1335a640 1 Processor -- start 2024-04-24T23:49:19.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.023+0000 7f6c1335a640 1 -- start start 2024-04-24T23:49:19.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.024+0000 7f6c1335a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6c0c0afc00 0x7f6c0c0b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.024+0000 7f6c1335a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6c0c0affe0 0x7f6c0c0b4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.024+0000 7f6c1335a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c0c0b6db0 con 0x7f6c0c0b68d0 2024-04-24T23:49:19.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.024+0000 7f6c1335a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f6c0c15c6a0 con 0x7f6c0c0afc00 2024-04-24T23:49:19.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.024+0000 7f6c1335a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6c0c15d420 con 0x7f6c0c0affe0 2024-04-24T23:49:19.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.025+0000 7f6c110f6640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6c0c0afc00 0x7f6c0c0b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.025+0000 7f6c108f5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6c0c0affe0 0x7f6c0c0b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.025+0000 7f6c118f7640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f6c0c0b68d0 0x7f6c0c15faf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:41200/0 (socket says 172.21.15.16:41200) 2024-04-24T23:49:19.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.025+0000 7f6c118f7640 1 -- 172.21.15.16:0/3329645655 learned_addr learned my addr 172.21.15.16:0/3329645655 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:19.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.025+0000 7f6c108f5640 1 -- 172.21.15.16:0/3329645655 >> v2:172.21.15.156:3300/0 conn(0x7f6c0c0afc00 msgr2=0x7f6c0c0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:19.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.025+0000 7f6c108f5640 1 --2- 172.21.15.16:0/3329645655 >> v2:172.21.15.156:3300/0 conn(0x7f6c0c0afc00 0x7f6c0c0b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.025+0000 7f6c108f5640 1 -- 172.21.15.16:0/3329645655 >> v1:172.21.15.156:6789/0 conn(0x7f6c0c0b68d0 legacy=0x7f6c0c15faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.025+0000 7f6c108f5640 1 -- 172.21.15.16:0/3329645655 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c15c940 con 0x7f6c0c0affe0 2024-04-24T23:49:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.025+0000 7f6c110f6640 1 --2- 172.21.15.16:0/3329645655 >> v2:172.21.15.156:3300/0 conn(0x7f6c0c0afc00 0x7f6c0c0b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.026+0000 7f6c108f5640 1 --2- 172.21.15.16:0/3329645655 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6c0c0affe0 0x7f6c0c0b4850 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0097a0 tx=0x7f6bfc02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1dd814dc3350e2f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.026+0000 7f6c03fff640 1 -- 172.21.15.16:0/3329645655 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc037070 con 0x7f6c0c0affe0 2024-04-24T23:49:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.026+0000 7f6c03fff640 1 -- 172.21.15.16:0/3329645655 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bfc032440 con 0x7f6c0c0affe0 2024-04-24T23:49:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.026+0000 7f6c03fff640 1 -- 172.21.15.16:0/3329645655 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc032810 con 0x7f6c0c0affe0 2024-04-24T23:49:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.026+0000 7f6c1335a640 1 -- 172.21.15.16:0/3329645655 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6c0c0affe0 msgr2=0x7f6c0c0b4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.026+0000 7f6c1335a640 1 --2- 172.21.15.16:0/3329645655 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6c0c0affe0 0x7f6c0c0b4850 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0097a0 tx=0x7f6bfc02a180 comp rx=0 tx=0).stop 2024-04-24T23:49:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.026+0000 7f6c1335a640 1 -- 172.21.15.16:0/3329645655 shutdown_connections 2024-04-24T23:49:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.026+0000 7f6c1335a640 1 --2- 172.21.15.16:0/3329645655 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6c0c0affe0 0x7f6c0c0b4850 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.026+0000 7f6c1335a640 1 --2- 172.21.15.16:0/3329645655 >> v2:172.21.15.156:3300/0 conn(0x7f6c0c0afc00 0x7f6c0c0b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.026+0000 7f6c1335a640 1 -- 172.21.15.16:0/3329645655 >> 172.21.15.16:0/3329645655 conn(0x7f6c0c071b40 msgr2=0x7f6c0c071f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.027+0000 7f6c1335a640 1 -- 172.21.15.16:0/3329645655 shutdown_connections 2024-04-24T23:49:19.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.027+0000 7f6c1335a640 1 -- 172.21.15.16:0/3329645655 wait complete. 2024-04-24T23:49:19.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.028+0000 7f6c1335a640 1 Processor -- start 2024-04-24T23:49:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.028+0000 7f6c1335a640 1 -- start start 2024-04-24T23:49:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.028+0000 7f6c1335a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c0afc00 0x7f6c0c141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.028+0000 7f6c1335a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6c0c0affe0 0x7f6c0c1e6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.028+0000 7f6c1335a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c0c0b68d0 0x7f6c0c0ab340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.028+0000 7f6c1335a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c0cfb10 con 0x7f6c0c0afc00 2024-04-24T23:49:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.028+0000 7f6c1335a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6c0c15cf00 con 0x7f6c0c0b68d0 2024-04-24T23:49:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c1335a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6c0c06edb0 con 0x7f6c0c0affe0 2024-04-24T23:49:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c110f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c0afc00 0x7f6c0c141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c110f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c0afc00 0x7f6c0c141f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48244/0 (socket says 172.21.15.16:48244) 2024-04-24T23:49:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c110f6640 1 -- 172.21.15.16:0/3548352079 learned_addr learned my addr 172.21.15.16:0/3548352079 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c118f7640 1 --2- 172.21.15.16:0/3548352079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c0c0b68d0 0x7f6c0c0ab340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c108f5640 1 --2- 172.21.15.16:0/3548352079 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6c0c0affe0 0x7f6c0c1e6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c110f6640 1 -- 172.21.15.16:0/3548352079 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6c0c0affe0 msgr2=0x7f6c0c1e6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c110f6640 1 --2- 172.21.15.16:0/3548352079 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6c0c0affe0 0x7f6c0c1e6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c110f6640 1 -- 172.21.15.16:0/3548352079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c0c0b68d0 msgr2=0x7f6c0c0ab340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c110f6640 1 --2- 172.21.15.16:0/3548352079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c0c0b68d0 0x7f6c0c0ab340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c110f6640 1 -- 172.21.15.16:0/3548352079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c0c0ad480 con 0x7f6c0c0afc00 2024-04-24T23:49:19.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c110f6640 1 --2- 172.21.15.16:0/3548352079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c0afc00 0x7f6c0c141f20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400ec10 tx=0x7f6c0400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:19.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c01ffb640 1 -- 172.21.15.16:0/3548352079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04017070 con 0x7f6c0c0afc00 2024-04-24T23:49:19.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c01ffb640 1 -- 172.21.15.16:0/3548352079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c0400cc30 con 0x7f6c0c0afc00 2024-04-24T23:49:19.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c1335a640 1 -- 172.21.15.16:0/3548352079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c0c1e6c70 con 0x7f6c0c0afc00 2024-04-24T23:49:19.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.029+0000 7f6c01ffb640 1 -- 172.21.15.16:0/3548352079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c04013470 con 0x7f6c0c0afc00 2024-04-24T23:49:19.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.030+0000 7f6c1335a640 1 -- 172.21.15.16:0/3548352079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c0c0ad720 con 0x7f6c0c0afc00 2024-04-24T23:49:19.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.031+0000 7f6c1335a640 1 -- 172.21.15.16:0/3548352079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c0c1e7190 con 0x7f6c0c0afc00 2024-04-24T23:49:19.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.031+0000 7f6c01ffb640 1 -- 172.21.15.16:0/3548352079 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6c0401a070 con 0x7f6c0c0afc00 2024-04-24T23:49:19.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.031+0000 7f6c01ffb640 1 --2- 172.21.15.16:0/3548352079 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6bf006be30 0x7f6bf006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.031+0000 7f6c01ffb640 1 -- 172.21.15.16:0/3548352079 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6c0408b400 con 0x7f6c0c0afc00 2024-04-24T23:49:19.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.031+0000 7f6c108f5640 1 --2- 172.21.15.16:0/3548352079 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6bf006be30 0x7f6bf006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.032+0000 7f6c108f5640 1 --2- 172.21.15.16:0/3548352079 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6bf006be30 0x7f6bf006e2f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc034000 tx=0x7f6bfc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:19.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.035+0000 7f6c01ffb640 1 -- 172.21.15.16:0/3548352079 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6c0c1e7190 con 0x7f6c0c0afc00 2024-04-24T23:49:19.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.221+0000 7f6c1335a640 1 -- 172.21.15.16:0/3548352079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c0c074a20 con 0x7f6c0c0afc00 2024-04-24T23:49:19.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.222+0000 7f6c01ffb640 1 -- 172.21.15.16:0/3548352079 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18502 (secure 0 0 0) 0x7f6c04059b00 con 0x7f6c0c0afc00 2024-04-24T23:49:19.222 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:19.222 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:59.888015+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:19.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.224+0000 7f6c1335a640 1 -- 172.21.15.16:0/3548352079 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6bf006be30 msgr2=0x7f6bf006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.224+0000 7f6c1335a640 1 --2- 172.21.15.16:0/3548352079 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6bf006be30 0x7f6bf006e2f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc034000 tx=0x7f6bfc02e000 comp rx=0 tx=0).stop 2024-04-24T23:49:19.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.224+0000 7f6c1335a640 1 -- 172.21.15.16:0/3548352079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c0afc00 msgr2=0x7f6c0c141f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.224+0000 7f6c1335a640 1 --2- 172.21.15.16:0/3548352079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c0afc00 0x7f6c0c141f20 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6c0400ec10 tx=0x7f6c0400c520 comp rx=0 tx=0).stop 2024-04-24T23:49:19.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.224+0000 7f6c1335a640 1 -- 172.21.15.16:0/3548352079 shutdown_connections 2024-04-24T23:49:19.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.224+0000 7f6c1335a640 1 --2- 172.21.15.16:0/3548352079 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6bf006be30 0x7f6bf006e2f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.224+0000 7f6c1335a640 1 --2- 172.21.15.16:0/3548352079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6c0c0b68d0 0x7f6c0c0ab340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.224+0000 7f6c1335a640 1 --2- 172.21.15.16:0/3548352079 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6c0c0affe0 0x7f6c0c1e6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.224+0000 7f6c1335a640 1 --2- 172.21.15.16:0/3548352079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c0c0afc00 0x7f6c0c141f20 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.224+0000 7f6c1335a640 1 -- 172.21.15.16:0/3548352079 >> 172.21.15.16:0/3548352079 conn(0x7f6c0c071b40 msgr2=0x7f6c0c0ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:19.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.224+0000 7f6c1335a640 1 -- 172.21.15.16:0/3548352079 shutdown_connections 2024-04-24T23:49:19.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.224+0000 7f6c1335a640 1 -- 172.21.15.16:0/3548352079 wait complete. 2024-04-24T23:49:19.235 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-24T23:49:19.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:49:19.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.336+0000 7f51f5342640 1 Processor -- start 2024-04-24T23:49:19.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.336+0000 7f51f5342640 1 -- start start 2024-04-24T23:49:19.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.337+0000 7f51f5342640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f51f015dfa0 0x7f51f00d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.337+0000 7f51f5342640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f51f00cdae0 0x7f51f00cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.337+0000 7f51f5342640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51f00d05a0 con 0x7f51f00ce5f0 2024-04-24T23:49:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.337+0000 7f51f5342640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f51f00b7360 con 0x7f51f00cdae0 2024-04-24T23:49:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.337+0000 7f51f5342640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f51f00cc640 con 0x7f51f015dfa0 2024-04-24T23:49:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.337+0000 7f51ef7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f51f00cdae0 0x7f51f00cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.337+0000 7f51ee7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f51f00ce5f0 0x7f51f00b5f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50588/0 (socket says 172.21.15.16:50588) 2024-04-24T23:49:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.337+0000 7f51ee7fc640 1 -- 172.21.15.16:0/814675525 learned_addr learned my addr 172.21.15.16:0/814675525 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.337+0000 7f51eeffd640 1 --2- 172.21.15.16:0/814675525 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f51f015dfa0 0x7f51f00d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.337+0000 7f51ef7fe640 1 -- 172.21.15.16:0/814675525 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f51f015dfa0 msgr2=0x7f51f00d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.337+0000 7f51ef7fe640 1 --2- 172.21.15.16:0/814675525 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f51f015dfa0 0x7f51f00d0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.337+0000 7f51ef7fe640 1 -- 172.21.15.16:0/814675525 >> v1:172.21.15.16:6789/0 conn(0x7f51f00ce5f0 legacy=0x7f51f00b5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.337+0000 7f51ef7fe640 1 -- 172.21.15.16:0/814675525 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f00d0880 con 0x7f51f00cdae0 2024-04-24T23:49:19.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.338+0000 7f51ef7fe640 1 --2- 172.21.15.16:0/814675525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f00cdae0 0x7f51f00cea70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f51f00cc7c0 tx=0x7f51e002a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2fd3790d0a50e4c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:19.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.338+0000 7f51edffb640 1 -- 172.21.15.16:0/814675525 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e002ae60 con 0x7f51f00cdae0 2024-04-24T23:49:19.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.338+0000 7f51edffb640 1 -- 172.21.15.16:0/814675525 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51e0007740 con 0x7f51f00cdae0 2024-04-24T23:49:19.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.338+0000 7f51edffb640 1 -- 172.21.15.16:0/814675525 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0007b10 con 0x7f51f00cdae0 2024-04-24T23:49:19.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.339+0000 7f51f5342640 1 -- 172.21.15.16:0/814675525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f00cdae0 msgr2=0x7f51f00cea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.339+0000 7f51f5342640 1 --2- 172.21.15.16:0/814675525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f00cdae0 0x7f51f00cea70 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f51f00cc7c0 tx=0x7f51e002a640 comp rx=0 tx=0).stop 2024-04-24T23:49:19.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.339+0000 7f51f5342640 1 -- 172.21.15.16:0/814675525 shutdown_connections 2024-04-24T23:49:19.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.339+0000 7f51f5342640 1 --2- 172.21.15.16:0/814675525 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f00cdae0 0x7f51f00cea70 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.339+0000 7f51f5342640 1 --2- 172.21.15.16:0/814675525 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f51f015dfa0 0x7f51f00d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.339+0000 7f51f5342640 1 -- 172.21.15.16:0/814675525 >> 172.21.15.16:0/814675525 conn(0x7f51f006b4d0 msgr2=0x7f51f00b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:19.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.339+0000 7f51f5342640 1 -- 172.21.15.16:0/814675525 shutdown_connections 2024-04-24T23:49:19.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.339+0000 7f51f5342640 1 -- 172.21.15.16:0/814675525 wait complete. 2024-04-24T23:49:19.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.340+0000 7f51f5342640 1 Processor -- start 2024-04-24T23:49:19.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.340+0000 7f51f5342640 1 -- start start 2024-04-24T23:49:19.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51f5342640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51f00cdae0 0x7f51f015c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51f5342640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f51f00ce5f0 0x7f51f01f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51f5342640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f015dfa0 0x7f51f00b2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51f5342640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f51f015e380 con 0x7f51f015dfa0 2024-04-24T23:49:19.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51f5342640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f51f00cc2d0 con 0x7f51f00cdae0 2024-04-24T23:49:19.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51f5342640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f51f00d1520 con 0x7f51f00ce5f0 2024-04-24T23:49:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51ef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f015dfa0 0x7f51f00b2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51ef7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f015dfa0 0x7f51f00b2230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48264/0 (socket says 172.21.15.16:48264) 2024-04-24T23:49:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51ef7fe640 1 -- 172.21.15.16:0/3972952218 learned_addr learned my addr 172.21.15.16:0/3972952218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51eeffd640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51f00cdae0 0x7f51f015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51ee7fc640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f51f00ce5f0 0x7f51f01f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51ef7fe640 1 -- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f51f00ce5f0 msgr2=0x7f51f01f4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51ef7fe640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f51f00ce5f0 0x7f51f01f4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51ef7fe640 1 -- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51f00cdae0 msgr2=0x7f51f015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51ef7fe640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51f00cdae0 0x7f51f015c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51ef7fe640 1 -- 172.21.15.16:0/3972952218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51f00aef00 con 0x7f51f015dfa0 2024-04-24T23:49:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51ee7fc640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f51f00ce5f0 0x7f51f01f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51eeffd640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51f00cdae0 0x7f51f015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.341+0000 7f51ef7fe640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f015dfa0 0x7f51f00b2230 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f51e002ac30 tx=0x7f51e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:19.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.342+0000 7f51e77fe640 1 -- 172.21.15.16:0/3972952218 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0040070 con 0x7f51f015dfa0 2024-04-24T23:49:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.342+0000 7f51e77fe640 1 -- 172.21.15.16:0/3972952218 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51e002ae60 con 0x7f51f015dfa0 2024-04-24T23:49:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.342+0000 7f51f5342640 1 -- 172.21.15.16:0/3972952218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51f00c6900 con 0x7f51f015dfa0 2024-04-24T23:49:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.342+0000 7f51e77fe640 1 -- 172.21.15.16:0/3972952218 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e0045460 con 0x7f51f015dfa0 2024-04-24T23:49:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.342+0000 7f51f5342640 1 -- 172.21.15.16:0/3972952218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51f00c6be0 con 0x7f51f015dfa0 2024-04-24T23:49:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.343+0000 7f51f5342640 1 -- 172.21.15.16:0/3972952218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51f006bea0 con 0x7f51f015dfa0 2024-04-24T23:49:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.343+0000 7f51e77fe640 1 -- 172.21.15.16:0/3972952218 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f51e0039040 con 0x7f51f015dfa0 2024-04-24T23:49:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.343+0000 7f51e77fe640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f51d006be60 0x7f51d006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.344+0000 7f51e77fe640 1 -- 172.21.15.16:0/3972952218 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f51e00ac990 con 0x7f51f015dfa0 2024-04-24T23:49:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.344+0000 7f51eeffd640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f51d006be60 0x7f51d006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.344+0000 7f51eeffd640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f51d006be60 0x7f51d006e320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f51f015cbf0 tx=0x7f51e8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:19.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.348+0000 7f51e77fe640 1 -- 172.21.15.16:0/3972952218 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f51e007ab90 con 0x7f51f015dfa0 2024-04-24T23:49:19.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.529+0000 7f51f5342640 1 -- 172.21.15.16:0/3972952218 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f51f015c720 con 0x7f51d006be60 2024-04-24T23:49:19.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.538+0000 7f51e77fe640 1 -- 172.21.15.16:0/3972952218 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f51f015c720 con 0x7f51d006be60 2024-04-24T23:49:19.542 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:19.544 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-24T23:49:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.541+0000 7f51f5342640 1 -- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f51d006be60 msgr2=0x7f51d006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.541+0000 7f51f5342640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f51d006be60 0x7f51d006e320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f51f015cbf0 tx=0x7f51e8008040 comp rx=0 tx=0).stop 2024-04-24T23:49:19.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.541+0000 7f51f5342640 1 -- 172.21.15.16:0/3972952218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f015dfa0 msgr2=0x7f51f00b2230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.541+0000 7f51f5342640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f015dfa0 0x7f51f00b2230 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f51e002ac30 tx=0x7f51e0004a40 comp rx=0 tx=0).stop 2024-04-24T23:49:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.542+0000 7f51f5342640 1 -- 172.21.15.16:0/3972952218 shutdown_connections 2024-04-24T23:49:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.542+0000 7f51f5342640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f51d006be60 0x7f51d006e320 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.542+0000 7f51f5342640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51f015dfa0 0x7f51f00b2230 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.542+0000 7f51f5342640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f51f00ce5f0 0x7f51f01f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.542+0000 7f51f5342640 1 --2- 172.21.15.16:0/3972952218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f51f00cdae0 0x7f51f015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.542+0000 7f51f5342640 1 -- 172.21.15.16:0/3972952218 >> 172.21.15.16:0/3972952218 conn(0x7f51f006b4d0 msgr2=0x7f51f00b6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.542+0000 7f51f5342640 1 -- 172.21.15.16:0/3972952218 shutdown_connections 2024-04-24T23:49:19.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.542+0000 7f51f5342640 1 -- 172.21.15.16:0/3972952218 wait complete. 2024-04-24T23:49:19.552 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-24T23:49:19.122640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248620,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754726100,"statfs":{"total":773094113280,"available":772839526400,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003107"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864269+0000","last_change":"2024-04-24T23:48:54.496506+0000","last_active":"2024-04-24T23:49:00.864269+0000","last_peered":"2024-04-24T23:49:00.864269+0000","last_clean":"2024-04-24T23:49:00.864269+0000","last_became_active":"2024-04-24T23:48:54.496184+0000","last_became_peered":"2024-04-24T23:48:54.496184+0000","last_unstale":"2024-04-24T23:49:00.864269+0000","last_undegraded":"2024-04-24T23:49:00.864269+0000","last_fullsized":"2024-04-24T23:49:00.864269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:26.511856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526502+0000","last_change":"2024-04-24T23:48:55.493918+0000","last_active":"2024-04-24T23:49:00.526502+0000","last_peered":"2024-04-24T23:49:00.526502+0000","last_clean":"2024-04-24T23:49:00.526502+0000","last_became_active":"2024-04-24T23:48:55.493534+0000","last_became_peered":"2024-04-24T23:48:55.493534+0000","last_unstale":"2024-04-24T23:49:00.526502+0000","last_undegraded":"2024-04-24T23:49:00.526502+0000","last_fullsized":"2024-04-24T23:49:00.526502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:56:24.335063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.525963+0000","last_change":"2024-04-24T23:48:59.535521+0000","last_active":"2024-04-24T23:49:00.525963+0000","last_peered":"2024-04-24T23:49:00.525963+0000","last_clean":"2024-04-24T23:49:00.525963+0000","last_became_active":"2024-04-24T23:48:59.535120+0000","last_became_peered":"2024-04-24T23:48:59.535120+0000","last_unstale":"2024-04-24T23:49:00.525963+0000","last_undegraded":"2024-04-24T23:49:00.525963+0000","last_fullsized":"2024-04-24T23:49:00.525963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:02:34.201356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526615+0000","last_change":"2024-04-24T23:48:54.494612+0000","last_active":"2024-04-24T23:49:00.526615+0000","last_peered":"2024-04-24T23:49:00.526615+0000","last_clean":"2024-04-24T23:49:00.526615+0000","last_became_active":"2024-04-24T23:48:54.494317+0000","last_became_peered":"2024-04-24T23:48:54.494317+0000","last_unstale":"2024-04-24T23:49:00.526615+0000","last_undegraded":"2024-04-24T23:49:00.526615+0000","last_fullsized":"2024-04-24T23:49:00.526615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:07.349136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864094+0000","last_change":"2024-04-24T23:48:55.512152+0000","last_active":"2024-04-24T23:49:00.864094+0000","last_peered":"2024-04-24T23:49:00.864094+0000","last_clean":"2024-04-24T23:49:00.864094+0000","last_became_active":"2024-04-24T23:48:55.511396+0000","last_became_peered":"2024-04-24T23:48:55.511396+0000","last_unstale":"2024-04-24T23:49:00.864094+0000","last_undegraded":"2024-04-24T23:49:00.864094+0000","last_fullsized":"2024-04-24T23:49:00.864094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510457+0000","last_change":"2024-04-24T23:48:59.530584+0000","last_active":"2024-04-24T23:49:00.510457+0000","last_peered":"2024-04-24T23:49:00.510457+0000","last_clean":"2024-04-24T23:49:00.510457+0000","last_became_active":"2024-04-24T23:48:59.530081+0000","last_became_peered":"2024-04-24T23:48:59.530081+0000","last_unstale":"2024-04-24T23:49:00.510457+0000","last_undegraded":"2024-04-24T23:49:00.510457+0000","last_fullsized":"2024-04-24T23:49:00.510457+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:03:32.497072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510851+0000","last_change":"2024-04-24T23:48:54.495374+0000","last_active":"2024-04-24T23:49:00.510851+0000","last_peered":"2024-04-24T23:49:00.510851+0000","last_clean":"2024-04-24T23:49:00.510851+0000","last_became_active":"2024-04-24T23:48:54.495040+0000","last_became_peered":"2024-04-24T23:48:54.495040+0000","last_unstale":"2024-04-24T23:49:00.510851+0000","last_undegraded":"2024-04-24T23:49:00.510851+0000","last_fullsized":"2024-04-24T23:49:00.510851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:31.412349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864681+0000","last_change":"2024-04-24T23:48:55.505099+0000","last_active":"2024-04-24T23:49:00.864681+0000","last_peered":"2024-04-24T23:49:00.864681+0000","last_clean":"2024-04-24T23:49:00.864681+0000","last_became_active":"2024-04-24T23:48:55.504778+0000","last_became_peered":"2024-04-24T23:48:55.504778+0000","last_unstale":"2024-04-24T23:49:00.864681+0000","last_undegraded":"2024-04-24T23:49:00.864681+0000","last_fullsized":"2024-04-24T23:49:00.864681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:17.488901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511291+0000","last_change":"2024-04-24T23:48:59.526537+0000","last_active":"2024-04-24T23:49:00.511291+0000","last_peered":"2024-04-24T23:49:00.511291+0000","last_clean":"2024-04-24T23:49:00.511291+0000","last_became_active":"2024-04-24T23:48:59.526038+0000","last_became_peered":"2024-04-24T23:48:59.526038+0000","last_unstale":"2024-04-24T23:49:00.511291+0000","last_undegraded":"2024-04-24T23:49:00.511291+0000","last_fullsized":"2024-04-24T23:49:00.511291+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510029+0000","last_change":"2024-04-24T23:48:54.498311+0000","last_active":"2024-04-24T23:49:00.510029+0000","last_peered":"2024-04-24T23:49:00.510029+0000","last_clean":"2024-04-24T23:49:00.510029+0000","last_became_active":"2024-04-24T23:48:54.497986+0000","last_became_peered":"2024-04-24T23:48:54.497986+0000","last_unstale":"2024-04-24T23:49:00.510029+0000","last_undegraded":"2024-04-24T23:49:00.510029+0000","last_fullsized":"2024-04-24T23:49:00.510029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865047+0000","last_change":"2024-04-24T23:48:55.507029+0000","last_active":"2024-04-24T23:49:00.865047+0000","last_peered":"2024-04-24T23:49:00.865047+0000","last_clean":"2024-04-24T23:49:00.865047+0000","last_became_active":"2024-04-24T23:48:55.506772+0000","last_became_peered":"2024-04-24T23:48:55.506772+0000","last_unstale":"2024-04-24T23:49:00.865047+0000","last_undegraded":"2024-04-24T23:49:00.865047+0000","last_fullsized":"2024-04-24T23:49:00.865047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:42.483054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509920+0000","last_change":"2024-04-24T23:48:59.530333+0000","last_active":"2024-04-24T23:49:00.509920+0000","last_peered":"2024-04-24T23:49:00.509920+0000","last_clean":"2024-04-24T23:49:00.509920+0000","last_became_active":"2024-04-24T23:48:59.529831+0000","last_became_peered":"2024-04-24T23:48:59.529831+0000","last_unstale":"2024-04-24T23:49:00.509920+0000","last_undegraded":"2024-04-24T23:49:00.509920+0000","last_fullsized":"2024-04-24T23:49:00.509920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:47.852102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864057+0000","last_change":"2024-04-24T23:48:55.504704+0000","last_active":"2024-04-24T23:49:00.864057+0000","last_peered":"2024-04-24T23:49:00.864057+0000","last_clean":"2024-04-24T23:49:00.864057+0000","last_became_active":"2024-04-24T23:48:55.504444+0000","last_became_peered":"2024-04-24T23:48:55.504444+0000","last_unstale":"2024-04-24T23:49:00.864057+0000","last_undegraded":"2024-04-24T23:49:00.864057+0000","last_fullsized":"2024-04-24T23:49:00.864057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510280+0000","last_change":"2024-04-24T23:48:54.493270+0000","last_active":"2024-04-24T23:49:00.510280+0000","last_peered":"2024-04-24T23:49:00.510280+0000","last_clean":"2024-04-24T23:49:00.510280+0000","last_became_active":"2024-04-24T23:48:54.492952+0000","last_became_peered":"2024-04-24T23:48:54.492952+0000","last_unstale":"2024-04-24T23:49:00.510280+0000","last_undegraded":"2024-04-24T23:49:00.510280+0000","last_fullsized":"2024-04-24T23:49:00.510280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:36:57.277988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510165+0000","last_change":"2024-04-24T23:48:59.530237+0000","last_active":"2024-04-24T23:49:00.510165+0000","last_peered":"2024-04-24T23:49:00.510165+0000","last_clean":"2024-04-24T23:49:00.510165+0000","last_became_active":"2024-04-24T23:48:59.529732+0000","last_became_peered":"2024-04-24T23:48:59.529732+0000","last_unstale":"2024-04-24T23:49:00.510165+0000","last_undegraded":"2024-04-24T23:49:00.510165+0000","last_fullsized":"2024-04-24T23:49:00.510165+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:59.783695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526172+0000","last_change":"2024-04-24T23:48:55.500061+0000","last_active":"2024-04-24T23:49:00.526172+0000","last_peered":"2024-04-24T23:49:00.526172+0000","last_clean":"2024-04-24T23:49:00.526172+0000","last_became_active":"2024-04-24T23:48:55.499515+0000","last_became_peered":"2024-04-24T23:48:55.499515+0000","last_unstale":"2024-04-24T23:49:00.526172+0000","last_undegraded":"2024-04-24T23:49:00.526172+0000","last_fullsized":"2024-04-24T23:49:00.526172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:10.235664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509506+0000","last_change":"2024-04-24T23:48:54.496643+0000","last_active":"2024-04-24T23:49:00.509506+0000","last_peered":"2024-04-24T23:49:00.509506+0000","last_clean":"2024-04-24T23:49:00.509506+0000","last_became_active":"2024-04-24T23:48:54.496367+0000","last_became_peered":"2024-04-24T23:48:54.496367+0000","last_unstale":"2024-04-24T23:49:00.509506+0000","last_undegraded":"2024-04-24T23:49:00.509506+0000","last_fullsized":"2024-04-24T23:49:00.509506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:17.375124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510348+0000","last_change":"2024-04-24T23:48:59.531327+0000","last_active":"2024-04-24T23:49:00.510348+0000","last_peered":"2024-04-24T23:49:00.510348+0000","last_clean":"2024-04-24T23:49:00.510348+0000","last_became_active":"2024-04-24T23:48:59.530959+0000","last_became_peered":"2024-04-24T23:48:59.530959+0000","last_unstale":"2024-04-24T23:49:00.510348+0000","last_undegraded":"2024-04-24T23:49:00.510348+0000","last_fullsized":"2024-04-24T23:49:00.510348+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:14.260616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526263+0000","last_change":"2024-04-24T23:48:55.494442+0000","last_active":"2024-04-24T23:49:00.526263+0000","last_peered":"2024-04-24T23:49:00.526263+0000","last_clean":"2024-04-24T23:49:00.526263+0000","last_became_active":"2024-04-24T23:48:55.494165+0000","last_became_peered":"2024-04-24T23:48:55.494165+0000","last_unstale":"2024-04-24T23:49:00.526263+0000","last_undegraded":"2024-04-24T23:49:00.526263+0000","last_fullsized":"2024-04-24T23:49:00.526263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508952+0000","last_change":"2024-04-24T23:48:54.488761+0000","last_active":"2024-04-24T23:49:00.508952+0000","last_peered":"2024-04-24T23:49:00.508952+0000","last_clean":"2024-04-24T23:49:00.508952+0000","last_became_active":"2024-04-24T23:48:54.488431+0000","last_became_peered":"2024-04-24T23:48:54.488431+0000","last_unstale":"2024-04-24T23:49:00.508952+0000","last_undegraded":"2024-04-24T23:49:00.508952+0000","last_fullsized":"2024-04-24T23:49:00.508952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509720+0000","last_change":"2024-04-24T23:48:59.523973+0000","last_active":"2024-04-24T23:49:00.509720+0000","last_peered":"2024-04-24T23:49:00.509720+0000","last_clean":"2024-04-24T23:49:00.509720+0000","last_became_active":"2024-04-24T23:48:59.520679+0000","last_became_peered":"2024-04-24T23:48:59.520679+0000","last_unstale":"2024-04-24T23:49:00.509720+0000","last_undegraded":"2024-04-24T23:49:00.509720+0000","last_fullsized":"2024-04-24T23:49:00.509720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511072+0000","last_change":"2024-04-24T23:48:55.505618+0000","last_active":"2024-04-24T23:49:00.511072+0000","last_peered":"2024-04-24T23:49:00.511072+0000","last_clean":"2024-04-24T23:49:00.511072+0000","last_became_active":"2024-04-24T23:48:55.504804+0000","last_became_peered":"2024-04-24T23:48:55.504804+0000","last_unstale":"2024-04-24T23:49:00.511072+0000","last_undegraded":"2024-04-24T23:49:00.511072+0000","last_fullsized":"2024-04-24T23:49:00.511072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:46.215368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509409+0000","last_change":"2024-04-24T23:48:54.496898+0000","last_active":"2024-04-24T23:49:00.509409+0000","last_peered":"2024-04-24T23:49:00.509409+0000","last_clean":"2024-04-24T23:49:00.509409+0000","last_became_active":"2024-04-24T23:48:54.496603+0000","last_became_peered":"2024-04-24T23:48:54.496603+0000","last_unstale":"2024-04-24T23:49:00.509409+0000","last_undegraded":"2024-04-24T23:49:00.509409+0000","last_fullsized":"2024-04-24T23:49:00.509409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510695+0000","last_change":"2024-04-24T23:48:59.533868+0000","last_active":"2024-04-24T23:49:00.510695+0000","last_peered":"2024-04-24T23:49:00.510695+0000","last_clean":"2024-04-24T23:49:00.510695+0000","last_became_active":"2024-04-24T23:48:59.533487+0000","last_became_peered":"2024-04-24T23:48:59.533487+0000","last_unstale":"2024-04-24T23:49:00.510695+0000","last_undegraded":"2024-04-24T23:49:00.510695+0000","last_fullsized":"2024-04-24T23:49:00.510695+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:21.513853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864071+0000","last_change":"2024-04-24T23:48:55.497734+0000","last_active":"2024-04-24T23:49:00.864071+0000","last_peered":"2024-04-24T23:49:00.864071+0000","last_clean":"2024-04-24T23:49:00.864071+0000","last_became_active":"2024-04-24T23:48:55.496634+0000","last_became_peered":"2024-04-24T23:48:55.496634+0000","last_unstale":"2024-04-24T23:49:00.864071+0000","last_undegraded":"2024-04-24T23:49:00.864071+0000","last_fullsized":"2024-04-24T23:49:00.864071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:56:24.335063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864834+0000","last_change":"2024-04-24T23:48:54.499625+0000","last_active":"2024-04-24T23:49:00.864834+0000","last_peered":"2024-04-24T23:49:00.864834+0000","last_clean":"2024-04-24T23:49:00.864834+0000","last_became_active":"2024-04-24T23:48:54.499357+0000","last_became_peered":"2024-04-24T23:48:54.499357+0000","last_unstale":"2024-04-24T23:49:00.864834+0000","last_undegraded":"2024-04-24T23:49:00.864834+0000","last_fullsized":"2024-04-24T23:49:00.864834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511165+0000","last_change":"2024-04-24T23:48:59.523817+0000","last_active":"2024-04-24T23:49:00.511165+0000","last_peered":"2024-04-24T23:49:00.511165+0000","last_clean":"2024-04-24T23:49:00.511165+0000","last_became_active":"2024-04-24T23:48:59.522505+0000","last_became_peered":"2024-04-24T23:48:59.522505+0000","last_unstale":"2024-04-24T23:49:00.511165+0000","last_undegraded":"2024-04-24T23:49:00.511165+0000","last_fullsized":"2024-04-24T23:49:00.511165+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:37.247560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526836+0000","last_change":"2024-04-24T23:48:55.490605+0000","last_active":"2024-04-24T23:49:00.526836+0000","last_peered":"2024-04-24T23:49:00.526836+0000","last_clean":"2024-04-24T23:49:00.526836+0000","last_became_active":"2024-04-24T23:48:55.490271+0000","last_became_peered":"2024-04-24T23:48:55.490271+0000","last_unstale":"2024-04-24T23:49:00.526836+0000","last_undegraded":"2024-04-24T23:49:00.526836+0000","last_fullsized":"2024-04-24T23:49:00.526836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:53:38.263114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509803+0000","last_change":"2024-04-24T23:48:54.493101+0000","last_active":"2024-04-24T23:49:00.509803+0000","last_peered":"2024-04-24T23:49:00.509803+0000","last_clean":"2024-04-24T23:49:00.509803+0000","last_became_active":"2024-04-24T23:48:54.492768+0000","last_became_peered":"2024-04-24T23:48:54.492768+0000","last_unstale":"2024-04-24T23:49:00.509803+0000","last_undegraded":"2024-04-24T23:49:00.509803+0000","last_fullsized":"2024-04-24T23:49:00.509803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:15:10.269984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.525949+0000","last_change":"2024-04-24T23:48:59.535686+0000","last_active":"2024-04-24T23:49:00.525949+0000","last_peered":"2024-04-24T23:49:00.525949+0000","last_clean":"2024-04-24T23:49:00.525949+0000","last_became_active":"2024-04-24T23:48:59.535320+0000","last_became_peered":"2024-04-24T23:48:59.535320+0000","last_unstale":"2024-04-24T23:49:00.525949+0000","last_undegraded":"2024-04-24T23:49:00.525949+0000","last_fullsized":"2024-04-24T23:49:00.525949+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:22.709044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509618+0000","last_change":"2024-04-24T23:48:55.493935+0000","last_active":"2024-04-24T23:49:00.509618+0000","last_peered":"2024-04-24T23:49:00.509618+0000","last_clean":"2024-04-24T23:49:00.509618+0000","last_became_active":"2024-04-24T23:48:55.493604+0000","last_became_peered":"2024-04-24T23:48:55.493604+0000","last_unstale":"2024-04-24T23:49:00.509618+0000","last_undegraded":"2024-04-24T23:49:00.509618+0000","last_fullsized":"2024-04-24T23:49:00.509618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510971+0000","last_change":"2024-04-24T23:48:54.497633+0000","last_active":"2024-04-24T23:49:00.510971+0000","last_peered":"2024-04-24T23:49:00.510971+0000","last_clean":"2024-04-24T23:49:00.510971+0000","last_became_active":"2024-04-24T23:48:54.497329+0000","last_became_peered":"2024-04-24T23:48:54.497329+0000","last_unstale":"2024-04-24T23:49:00.510971+0000","last_undegraded":"2024-04-24T23:49:00.510971+0000","last_fullsized":"2024-04-24T23:49:00.510971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:37.806706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864823+0000","last_change":"2024-04-24T23:48:59.538720+0000","last_active":"2024-04-24T23:49:00.864823+0000","last_peered":"2024-04-24T23:49:00.864823+0000","last_clean":"2024-04-24T23:49:00.864823+0000","last_became_active":"2024-04-24T23:48:59.538310+0000","last_became_peered":"2024-04-24T23:48:59.538310+0000","last_unstale":"2024-04-24T23:49:00.864823+0000","last_undegraded":"2024-04-24T23:49:00.864823+0000","last_fullsized":"2024-04-24T23:49:00.864823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:10.292844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865110+0000","last_change":"2024-04-24T23:48:55.510066+0000","last_active":"2024-04-24T23:49:00.865110+0000","last_peered":"2024-04-24T23:49:00.865110+0000","last_clean":"2024-04-24T23:49:00.865110+0000","last_became_active":"2024-04-24T23:48:55.507977+0000","last_became_peered":"2024-04-24T23:48:55.507977+0000","last_unstale":"2024-04-24T23:49:00.865110+0000","last_undegraded":"2024-04-24T23:49:00.865110+0000","last_fullsized":"2024-04-24T23:49:00.865110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:06.267893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509767+0000","last_change":"2024-04-24T23:48:54.493032+0000","last_active":"2024-04-24T23:49:00.509767+0000","last_peered":"2024-04-24T23:49:00.509767+0000","last_clean":"2024-04-24T23:49:00.509767+0000","last_became_active":"2024-04-24T23:48:54.492694+0000","last_became_peered":"2024-04-24T23:48:54.492694+0000","last_unstale":"2024-04-24T23:49:00.509767+0000","last_undegraded":"2024-04-24T23:49:00.509767+0000","last_fullsized":"2024-04-24T23:49:00.509767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509213+0000","last_change":"2024-04-24T23:48:59.527964+0000","last_active":"2024-04-24T23:49:00.509213+0000","last_peered":"2024-04-24T23:49:00.509213+0000","last_clean":"2024-04-24T23:49:00.509213+0000","last_became_active":"2024-04-24T23:48:59.527519+0000","last_became_peered":"2024-04-24T23:48:59.527519+0000","last_unstale":"2024-04-24T23:49:00.509213+0000","last_undegraded":"2024-04-24T23:49:00.509213+0000","last_fullsized":"2024-04-24T23:49:00.509213+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:37.247560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511053+0000","last_change":"2024-04-24T23:48:55.505703+0000","last_active":"2024-04-24T23:49:00.511053+0000","last_peered":"2024-04-24T23:49:00.511053+0000","last_clean":"2024-04-24T23:49:00.511053+0000","last_became_active":"2024-04-24T23:48:55.505257+0000","last_became_peered":"2024-04-24T23:48:55.505257+0000","last_unstale":"2024-04-24T23:49:00.511053+0000","last_undegraded":"2024-04-24T23:49:00.511053+0000","last_fullsized":"2024-04-24T23:49:00.511053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:41.372867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864734+0000","last_change":"2024-04-24T23:48:54.494961+0000","last_active":"2024-04-24T23:49:00.864734+0000","last_peered":"2024-04-24T23:49:00.864734+0000","last_clean":"2024-04-24T23:49:00.864734+0000","last_became_active":"2024-04-24T23:48:54.494672+0000","last_became_peered":"2024-04-24T23:48:54.494672+0000","last_unstale":"2024-04-24T23:49:00.864734+0000","last_undegraded":"2024-04-24T23:49:00.864734+0000","last_fullsized":"2024-04-24T23:49:00.864734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509770+0000","last_change":"2024-04-24T23:48:59.525981+0000","last_active":"2024-04-24T23:49:00.509770+0000","last_peered":"2024-04-24T23:49:00.509770+0000","last_clean":"2024-04-24T23:49:00.509770+0000","last_became_active":"2024-04-24T23:48:59.524788+0000","last_became_peered":"2024-04-24T23:48:59.524788+0000","last_unstale":"2024-04-24T23:49:00.509770+0000","last_undegraded":"2024-04-24T23:49:00.509770+0000","last_fullsized":"2024-04-24T23:49:00.509770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:08.826500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508934+0000","last_change":"2024-04-24T23:48:55.493966+0000","last_active":"2024-04-24T23:49:00.508934+0000","last_peered":"2024-04-24T23:49:00.508934+0000","last_clean":"2024-04-24T23:49:00.508934+0000","last_became_active":"2024-04-24T23:48:55.493622+0000","last_became_peered":"2024-04-24T23:48:55.493622+0000","last_unstale":"2024-04-24T23:49:00.508934+0000","last_undegraded":"2024-04-24T23:49:00.508934+0000","last_fullsized":"2024-04-24T23:49:00.508934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863565+0000","last_change":"2024-04-24T23:48:54.497811+0000","last_active":"2024-04-24T23:49:00.863565+0000","last_peered":"2024-04-24T23:49:00.863565+0000","last_clean":"2024-04-24T23:49:00.863565+0000","last_became_active":"2024-04-24T23:48:54.497465+0000","last_became_peered":"2024-04-24T23:48:54.497465+0000","last_unstale":"2024-04-24T23:49:00.863565+0000","last_undegraded":"2024-04-24T23:49:00.863565+0000","last_fullsized":"2024-04-24T23:49:00.863565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:17.375124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509953+0000","last_change":"2024-04-24T23:48:59.536596+0000","last_active":"2024-04-24T23:49:00.509953+0000","last_peered":"2024-04-24T23:49:00.509953+0000","last_clean":"2024-04-24T23:49:00.509953+0000","last_became_active":"2024-04-24T23:48:59.536186+0000","last_became_peered":"2024-04-24T23:48:59.536186+0000","last_unstale":"2024-04-24T23:49:00.509953+0000","last_undegraded":"2024-04-24T23:49:00.509953+0000","last_fullsized":"2024-04-24T23:49:00.509953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:46.508006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863695+0000","last_change":"2024-04-24T23:48:55.507490+0000","last_active":"2024-04-24T23:49:00.863695+0000","last_peered":"2024-04-24T23:49:00.863695+0000","last_clean":"2024-04-24T23:49:00.863695+0000","last_became_active":"2024-04-24T23:48:55.506415+0000","last_became_peered":"2024-04-24T23:48:55.506415+0000","last_unstale":"2024-04-24T23:49:00.863695+0000","last_undegraded":"2024-04-24T23:49:00.863695+0000","last_fullsized":"2024-04-24T23:49:00.863695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509313+0000","last_change":"2024-04-24T23:48:54.491388+0000","last_active":"2024-04-24T23:49:00.509313+0000","last_peered":"2024-04-24T23:49:00.509313+0000","last_clean":"2024-04-24T23:49:00.509313+0000","last_became_active":"2024-04-24T23:48:54.490991+0000","last_became_peered":"2024-04-24T23:48:54.490991+0000","last_unstale":"2024-04-24T23:49:00.509313+0000","last_undegraded":"2024-04-24T23:49:00.509313+0000","last_fullsized":"2024-04-24T23:49:00.509313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863944+0000","last_change":"2024-04-24T23:48:59.539922+0000","last_active":"2024-04-24T23:49:00.863944+0000","last_peered":"2024-04-24T23:49:00.863944+0000","last_clean":"2024-04-24T23:49:00.863944+0000","last_became_active":"2024-04-24T23:48:59.539449+0000","last_became_peered":"2024-04-24T23:48:59.539449+0000","last_unstale":"2024-04-24T23:49:00.863944+0000","last_undegraded":"2024-04-24T23:49:00.863944+0000","last_fullsized":"2024-04-24T23:49:00.863944+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:08.826500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526078+0000","last_change":"2024-04-24T23:48:55.509237+0000","last_active":"2024-04-24T23:49:00.526078+0000","last_peered":"2024-04-24T23:49:00.526078+0000","last_clean":"2024-04-24T23:49:00.526078+0000","last_became_active":"2024-04-24T23:48:55.508847+0000","last_became_peered":"2024-04-24T23:48:55.508847+0000","last_unstale":"2024-04-24T23:49:00.526078+0000","last_undegraded":"2024-04-24T23:49:00.526078+0000","last_fullsized":"2024-04-24T23:49:00.526078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864955+0000","last_change":"2024-04-24T23:48:54.496310+0000","last_active":"2024-04-24T23:49:00.864955+0000","last_peered":"2024-04-24T23:49:00.864955+0000","last_clean":"2024-04-24T23:49:00.864955+0000","last_became_active":"2024-04-24T23:48:54.496016+0000","last_became_peered":"2024-04-24T23:48:54.496016+0000","last_unstale":"2024-04-24T23:49:00.864955+0000","last_undegraded":"2024-04-24T23:49:00.864955+0000","last_fullsized":"2024-04-24T23:49:00.864955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509522+0000","last_change":"2024-04-24T23:48:45.082459+0000","last_active":"2024-04-24T23:49:00.509522+0000","last_peered":"2024-04-24T23:49:00.509522+0000","last_clean":"2024-04-24T23:49:00.509522+0000","last_became_active":"2024-04-24T23:48:45.081495+0000","last_became_peered":"2024-04-24T23:48:45.081495+0000","last_unstale":"2024-04-24T23:49:00.509522+0000","last_undegraded":"2024-04-24T23:49:00.509522+0000","last_fullsized":"2024-04-24T23:49:00.509522+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_clean_scrub_stamp":"2024-04-24T23:48:43.440993+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:53:20.832511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864431+0000","last_change":"2024-04-24T23:48:59.521532+0000","last_active":"2024-04-24T23:49:00.864431+0000","last_peered":"2024-04-24T23:49:00.864431+0000","last_clean":"2024-04-24T23:49:00.864431+0000","last_became_active":"2024-04-24T23:48:59.521144+0000","last_became_peered":"2024-04-24T23:48:59.521144+0000","last_unstale":"2024-04-24T23:49:00.864431+0000","last_undegraded":"2024-04-24T23:49:00.864431+0000","last_fullsized":"2024-04-24T23:49:00.864431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:08.826500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510601+0000","last_change":"2024-04-24T23:48:55.509123+0000","last_active":"2024-04-24T23:49:00.510601+0000","last_peered":"2024-04-24T23:49:00.510601+0000","last_clean":"2024-04-24T23:49:00.510601+0000","last_became_active":"2024-04-24T23:48:55.508860+0000","last_became_peered":"2024-04-24T23:48:55.508860+0000","last_unstale":"2024-04-24T23:49:00.510601+0000","last_undegraded":"2024-04-24T23:49:00.510601+0000","last_fullsized":"2024-04-24T23:49:00.510601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:03:28.472120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863884+0000","last_change":"2024-04-24T23:48:54.498741+0000","last_active":"2024-04-24T23:49:00.863884+0000","last_peered":"2024-04-24T23:49:00.863884+0000","last_clean":"2024-04-24T23:49:00.863884+0000","last_became_active":"2024-04-24T23:48:54.498241+0000","last_became_peered":"2024-04-24T23:48:54.498241+0000","last_unstale":"2024-04-24T23:49:00.863884+0000","last_undegraded":"2024-04-24T23:49:00.863884+0000","last_fullsized":"2024-04-24T23:49:00.863884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:07.349136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864020+0000","last_change":"2024-04-24T23:48:59.528434+0000","last_active":"2024-04-24T23:49:00.864020+0000","last_peered":"2024-04-24T23:49:00.864020+0000","last_clean":"2024-04-24T23:49:00.864020+0000","last_became_active":"2024-04-24T23:48:59.528062+0000","last_became_peered":"2024-04-24T23:48:59.528062+0000","last_unstale":"2024-04-24T23:49:00.864020+0000","last_undegraded":"2024-04-24T23:49:00.864020+0000","last_fullsized":"2024-04-24T23:49:00.864020+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864801+0000","last_change":"2024-04-24T23:48:55.509799+0000","last_active":"2024-04-24T23:49:00.864801+0000","last_peered":"2024-04-24T23:49:00.864801+0000","last_clean":"2024-04-24T23:49:00.864801+0000","last_became_active":"2024-04-24T23:48:55.509286+0000","last_became_peered":"2024-04-24T23:48:55.509286+0000","last_unstale":"2024-04-24T23:49:00.864801+0000","last_undegraded":"2024-04-24T23:49:00.864801+0000","last_fullsized":"2024-04-24T23:49:00.864801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:33.222608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509751+0000","last_change":"2024-04-24T23:48:54.490614+0000","last_active":"2024-04-24T23:49:00.509751+0000","last_peered":"2024-04-24T23:49:00.509751+0000","last_clean":"2024-04-24T23:49:00.509751+0000","last_became_active":"2024-04-24T23:48:54.490029+0000","last_became_peered":"2024-04-24T23:48:54.490029+0000","last_unstale":"2024-04-24T23:49:00.509751+0000","last_undegraded":"2024-04-24T23:49:00.509751+0000","last_fullsized":"2024-04-24T23:49:00.509751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864962+0000","last_change":"2024-04-24T23:48:59.529966+0000","last_active":"2024-04-24T23:49:00.864962+0000","last_peered":"2024-04-24T23:49:00.864962+0000","last_clean":"2024-04-24T23:49:00.864962+0000","last_became_active":"2024-04-24T23:48:59.529508+0000","last_became_peered":"2024-04-24T23:48:59.529508+0000","last_unstale":"2024-04-24T23:49:00.864962+0000","last_undegraded":"2024-04-24T23:49:00.864962+0000","last_fullsized":"2024-04-24T23:49:00.864962+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:46.508006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511361+0000","last_change":"2024-04-24T23:48:55.506681+0000","last_active":"2024-04-24T23:49:00.511361+0000","last_peered":"2024-04-24T23:49:00.511361+0000","last_clean":"2024-04-24T23:49:00.511361+0000","last_became_active":"2024-04-24T23:48:55.506393+0000","last_became_peered":"2024-04-24T23:48:55.506393+0000","last_unstale":"2024-04-24T23:49:00.511361+0000","last_undegraded":"2024-04-24T23:49:00.511361+0000","last_fullsized":"2024-04-24T23:49:00.511361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:02:30.176405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863990+0000","last_change":"2024-04-24T23:48:54.499321+0000","last_active":"2024-04-24T23:49:00.863990+0000","last_peered":"2024-04-24T23:49:00.863990+0000","last_clean":"2024-04-24T23:49:00.863990+0000","last_became_active":"2024-04-24T23:48:54.499015+0000","last_became_peered":"2024-04-24T23:48:54.499015+0000","last_unstale":"2024-04-24T23:49:00.863990+0000","last_undegraded":"2024-04-24T23:49:00.863990+0000","last_fullsized":"2024-04-24T23:49:00.863990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863856+0000","last_change":"2024-04-24T23:48:59.522405+0000","last_active":"2024-04-24T23:49:00.863856+0000","last_peered":"2024-04-24T23:49:00.863856+0000","last_clean":"2024-04-24T23:49:00.863856+0000","last_became_active":"2024-04-24T23:48:59.521434+0000","last_became_peered":"2024-04-24T23:48:59.521434+0000","last_unstale":"2024-04-24T23:49:00.863856+0000","last_undegraded":"2024-04-24T23:49:00.863856+0000","last_fullsized":"2024-04-24T23:49:00.863856+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:39.512436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511297+0000","last_change":"2024-04-24T23:48:55.507123+0000","last_active":"2024-04-24T23:49:00.511297+0000","last_peered":"2024-04-24T23:49:00.511297+0000","last_clean":"2024-04-24T23:49:00.511297+0000","last_became_active":"2024-04-24T23:48:55.506838+0000","last_became_peered":"2024-04-24T23:48:55.506838+0000","last_unstale":"2024-04-24T23:49:00.511297+0000","last_undegraded":"2024-04-24T23:49:00.511297+0000","last_fullsized":"2024-04-24T23:49:00.511297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:17.488901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509852+0000","last_change":"2024-04-24T23:48:54.494375+0000","last_active":"2024-04-24T23:49:00.509852+0000","last_peered":"2024-04-24T23:49:00.509852+0000","last_clean":"2024-04-24T23:49:00.509852+0000","last_became_active":"2024-04-24T23:48:54.494080+0000","last_became_peered":"2024-04-24T23:48:54.494080+0000","last_unstale":"2024-04-24T23:49:00.509852+0000","last_undegraded":"2024-04-24T23:49:00.509852+0000","last_fullsized":"2024-04-24T23:49:00.509852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:46.268530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510042+0000","last_change":"2024-04-24T23:48:59.530805+0000","last_active":"2024-04-24T23:49:00.510042+0000","last_peered":"2024-04-24T23:49:00.510042+0000","last_clean":"2024-04-24T23:49:00.510042+0000","last_became_active":"2024-04-24T23:48:59.530126+0000","last_became_peered":"2024-04-24T23:48:59.530126+0000","last_unstale":"2024-04-24T23:49:00.510042+0000","last_undegraded":"2024-04-24T23:49:00.510042+0000","last_fullsized":"2024-04-24T23:49:00.510042+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:50.914824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509597+0000","last_change":"2024-04-24T23:48:55.499881+0000","last_active":"2024-04-24T23:49:00.509597+0000","last_peered":"2024-04-24T23:49:00.509597+0000","last_clean":"2024-04-24T23:49:00.509597+0000","last_became_active":"2024-04-24T23:48:55.499493+0000","last_became_peered":"2024-04-24T23:48:55.499493+0000","last_unstale":"2024-04-24T23:49:00.509597+0000","last_undegraded":"2024-04-24T23:49:00.509597+0000","last_fullsized":"2024-04-24T23:49:00.509597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:33.222608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863280+0000","last_change":"2024-04-24T23:48:54.493205+0000","last_active":"2024-04-24T23:49:00.863280+0000","last_peered":"2024-04-24T23:49:00.863280+0000","last_clean":"2024-04-24T23:49:00.863280+0000","last_became_active":"2024-04-24T23:48:54.492446+0000","last_became_peered":"2024-04-24T23:48:54.492446+0000","last_unstale":"2024-04-24T23:49:00.863280+0000","last_undegraded":"2024-04-24T23:49:00.863280+0000","last_fullsized":"2024-04-24T23:49:00.863280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:16:25.658826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863935+0000","last_change":"2024-04-24T23:48:59.532383+0000","last_active":"2024-04-24T23:49:00.863935+0000","last_peered":"2024-04-24T23:49:00.863935+0000","last_clean":"2024-04-24T23:49:00.863935+0000","last_became_active":"2024-04-24T23:48:59.529606+0000","last_became_peered":"2024-04-24T23:48:59.529606+0000","last_unstale":"2024-04-24T23:49:00.863935+0000","last_undegraded":"2024-04-24T23:49:00.863935+0000","last_fullsized":"2024-04-24T23:49:00.863935+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864681+0000","last_change":"2024-04-24T23:48:55.499374+0000","last_active":"2024-04-24T23:49:00.864681+0000","last_peered":"2024-04-24T23:49:00.864681+0000","last_clean":"2024-04-24T23:49:00.864681+0000","last_became_active":"2024-04-24T23:48:55.499107+0000","last_became_peered":"2024-04-24T23:48:55.499107+0000","last_unstale":"2024-04-24T23:49:00.864681+0000","last_undegraded":"2024-04-24T23:49:00.864681+0000","last_fullsized":"2024-04-24T23:49:00.864681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864217+0000","last_change":"2024-04-24T23:48:54.491505+0000","last_active":"2024-04-24T23:49:00.864217+0000","last_peered":"2024-04-24T23:49:00.864217+0000","last_clean":"2024-04-24T23:49:00.864217+0000","last_became_active":"2024-04-24T23:48:54.491230+0000","last_became_peered":"2024-04-24T23:48:54.491230+0000","last_unstale":"2024-04-24T23:49:00.864217+0000","last_undegraded":"2024-04-24T23:49:00.864217+0000","last_fullsized":"2024-04-24T23:49:00.864217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526800+0000","last_change":"2024-04-24T23:48:59.531330+0000","last_active":"2024-04-24T23:49:00.526800+0000","last_peered":"2024-04-24T23:49:00.526800+0000","last_clean":"2024-04-24T23:49:00.526800+0000","last_became_active":"2024-04-24T23:48:59.530916+0000","last_became_peered":"2024-04-24T23:48:59.530916+0000","last_unstale":"2024-04-24T23:49:00.526800+0000","last_undegraded":"2024-04-24T23:49:00.526800+0000","last_fullsized":"2024-04-24T23:49:00.526800+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:43.024671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510174+0000","last_change":"2024-04-24T23:48:55.502184+0000","last_active":"2024-04-24T23:49:00.510174+0000","last_peered":"2024-04-24T23:49:00.510174+0000","last_clean":"2024-04-24T23:49:00.510174+0000","last_became_active":"2024-04-24T23:48:55.501682+0000","last_became_peered":"2024-04-24T23:48:55.501682+0000","last_unstale":"2024-04-24T23:49:00.510174+0000","last_undegraded":"2024-04-24T23:49:00.510174+0000","last_fullsized":"2024-04-24T23:49:00.510174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:06.267893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864548+0000","last_change":"2024-04-24T23:48:54.499017+0000","last_active":"2024-04-24T23:49:00.864548+0000","last_peered":"2024-04-24T23:49:00.864548+0000","last_clean":"2024-04-24T23:49:00.864548+0000","last_became_active":"2024-04-24T23:48:54.498747+0000","last_became_peered":"2024-04-24T23:48:54.498747+0000","last_unstale":"2024-04-24T23:49:00.864548+0000","last_undegraded":"2024-04-24T23:49:00.864548+0000","last_fullsized":"2024-04-24T23:49:00.864548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526541+0000","last_change":"2024-04-24T23:48:59.529890+0000","last_active":"2024-04-24T23:49:00.526541+0000","last_peered":"2024-04-24T23:49:00.526541+0000","last_clean":"2024-04-24T23:49:00.526541+0000","last_became_active":"2024-04-24T23:48:59.529534+0000","last_became_peered":"2024-04-24T23:48:59.529534+0000","last_unstale":"2024-04-24T23:49:00.526541+0000","last_undegraded":"2024-04-24T23:49:00.526541+0000","last_fullsized":"2024-04-24T23:49:00.526541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:38:25.945723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864543+0000","last_change":"2024-04-24T23:48:55.497917+0000","last_active":"2024-04-24T23:49:00.864543+0000","last_peered":"2024-04-24T23:49:00.864543+0000","last_clean":"2024-04-24T23:49:00.864543+0000","last_became_active":"2024-04-24T23:48:55.497158+0000","last_became_peered":"2024-04-24T23:48:55.497158+0000","last_unstale":"2024-04-24T23:49:00.864543+0000","last_undegraded":"2024-04-24T23:49:00.864543+0000","last_fullsized":"2024-04-24T23:49:00.864543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:51.930595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865010+0000","last_change":"2024-04-24T23:48:54.491006+0000","last_active":"2024-04-24T23:49:00.865010+0000","last_peered":"2024-04-24T23:49:00.865010+0000","last_clean":"2024-04-24T23:49:00.865010+0000","last_became_active":"2024-04-24T23:48:54.489661+0000","last_became_peered":"2024-04-24T23:48:54.489661+0000","last_unstale":"2024-04-24T23:49:00.865010+0000","last_undegraded":"2024-04-24T23:49:00.865010+0000","last_fullsized":"2024-04-24T23:49:00.865010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:16:25.658826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864966+0000","last_change":"2024-04-24T23:48:59.521225+0000","last_active":"2024-04-24T23:49:00.864966+0000","last_peered":"2024-04-24T23:49:00.864966+0000","last_clean":"2024-04-24T23:49:00.864966+0000","last_became_active":"2024-04-24T23:48:59.520761+0000","last_became_peered":"2024-04-24T23:48:59.520761+0000","last_unstale":"2024-04-24T23:49:00.864966+0000","last_undegraded":"2024-04-24T23:49:00.864966+0000","last_fullsized":"2024-04-24T23:49:00.864966+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:38:25.945723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510500+0000","last_change":"2024-04-24T23:48:55.487564+0000","last_active":"2024-04-24T23:49:00.510500+0000","last_peered":"2024-04-24T23:49:00.510500+0000","last_clean":"2024-04-24T23:49:00.510500+0000","last_became_active":"2024-04-24T23:48:55.487272+0000","last_became_peered":"2024-04-24T23:48:55.487272+0000","last_unstale":"2024-04-24T23:49:00.510500+0000","last_undegraded":"2024-04-24T23:49:00.510500+0000","last_fullsized":"2024-04-24T23:49:00.510500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:10.235664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:04.927990+0000","last_change":"2024-04-24T23:48:54.500818+0000","last_active":"2024-04-24T23:49:04.927990+0000","last_peered":"2024-04-24T23:49:04.927990+0000","last_clean":"2024-04-24T23:49:04.927990+0000","last_became_active":"2024-04-24T23:48:54.500560+0000","last_became_peered":"2024-04-24T23:48:54.500560+0000","last_unstale":"2024-04-24T23:49:04.927990+0000","last_undegraded":"2024-04-24T23:49:04.927990+0000","last_fullsized":"2024-04-24T23:49:04.927990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509881+0000","last_change":"2024-04-24T23:48:59.530292+0000","last_active":"2024-04-24T23:49:00.509881+0000","last_peered":"2024-04-24T23:49:00.509881+0000","last_clean":"2024-04-24T23:49:00.509881+0000","last_became_active":"2024-04-24T23:48:59.529429+0000","last_became_peered":"2024-04-24T23:48:59.529429+0000","last_unstale":"2024-04-24T23:49:00.509881+0000","last_undegraded":"2024-04-24T23:49:00.509881+0000","last_fullsized":"2024-04-24T23:49:00.509881+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:10.292844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510600+0000","last_change":"2024-04-24T23:48:55.502480+0000","last_active":"2024-04-24T23:49:00.510600+0000","last_peered":"2024-04-24T23:49:00.510600+0000","last_clean":"2024-04-24T23:49:00.510600+0000","last_became_active":"2024-04-24T23:48:55.502104+0000","last_became_peered":"2024-04-24T23:48:55.502104+0000","last_unstale":"2024-04-24T23:49:00.510600+0000","last_undegraded":"2024-04-24T23:49:00.510600+0000","last_fullsized":"2024-04-24T23:49:00.510600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:25.555824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509534+0000","last_change":"2024-04-24T23:48:54.495473+0000","last_active":"2024-04-24T23:49:00.509534+0000","last_peered":"2024-04-24T23:49:00.509534+0000","last_clean":"2024-04-24T23:49:00.509534+0000","last_became_active":"2024-04-24T23:48:54.495173+0000","last_became_peered":"2024-04-24T23:48:54.495173+0000","last_unstale":"2024-04-24T23:49:00.509534+0000","last_undegraded":"2024-04-24T23:49:00.509534+0000","last_fullsized":"2024-04-24T23:49:00.509534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510914+0000","last_change":"2024-04-24T23:48:59.532057+0000","last_active":"2024-04-24T23:49:00.510914+0000","last_peered":"2024-04-24T23:49:00.510914+0000","last_clean":"2024-04-24T23:49:00.510914+0000","last_became_active":"2024-04-24T23:48:59.530378+0000","last_became_peered":"2024-04-24T23:48:59.530378+0000","last_unstale":"2024-04-24T23:49:00.510914+0000","last_undegraded":"2024-04-24T23:49:00.510914+0000","last_fullsized":"2024-04-24T23:49:00.510914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:51.652134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864215+0000","last_change":"2024-04-24T23:48:55.508829+0000","last_active":"2024-04-24T23:49:00.864215+0000","last_peered":"2024-04-24T23:49:00.864215+0000","last_clean":"2024-04-24T23:49:00.864215+0000","last_became_active":"2024-04-24T23:48:55.508042+0000","last_became_peered":"2024-04-24T23:48:55.508042+0000","last_unstale":"2024-04-24T23:49:00.864215+0000","last_undegraded":"2024-04-24T23:49:00.864215+0000","last_fullsized":"2024-04-24T23:49:00.864215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865051+0000","last_change":"2024-04-24T23:48:54.497886+0000","last_active":"2024-04-24T23:49:00.865051+0000","last_peered":"2024-04-24T23:49:00.865051+0000","last_clean":"2024-04-24T23:49:00.865051+0000","last_became_active":"2024-04-24T23:48:54.497630+0000","last_became_peered":"2024-04-24T23:48:54.497630+0000","last_unstale":"2024-04-24T23:49:00.865051+0000","last_undegraded":"2024-04-24T23:49:00.865051+0000","last_fullsized":"2024-04-24T23:49:00.865051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:31.412349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526184+0000","last_change":"2024-04-24T23:48:59.532080+0000","last_active":"2024-04-24T23:49:00.526184+0000","last_peered":"2024-04-24T23:49:00.526184+0000","last_clean":"2024-04-24T23:49:00.526184+0000","last_became_active":"2024-04-24T23:48:59.531298+0000","last_became_peered":"2024-04-24T23:48:59.531298+0000","last_unstale":"2024-04-24T23:49:00.526184+0000","last_undegraded":"2024-04-24T23:49:00.526184+0000","last_fullsized":"2024-04-24T23:49:00.526184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:39.512436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864909+0000","last_change":"2024-04-24T23:48:55.513313+0000","last_active":"2024-04-24T23:49:00.864909+0000","last_peered":"2024-04-24T23:49:00.864909+0000","last_clean":"2024-04-24T23:49:00.864909+0000","last_became_active":"2024-04-24T23:48:55.513021+0000","last_became_peered":"2024-04-24T23:48:55.513021+0000","last_unstale":"2024-04-24T23:49:00.864909+0000","last_undegraded":"2024-04-24T23:49:00.864909+0000","last_fullsized":"2024-04-24T23:49:00.864909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:04.801548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864612+0000","last_change":"2024-04-24T23:48:54.498900+0000","last_active":"2024-04-24T23:49:00.864612+0000","last_peered":"2024-04-24T23:49:00.864612+0000","last_clean":"2024-04-24T23:49:00.864612+0000","last_became_active":"2024-04-24T23:48:54.498604+0000","last_became_peered":"2024-04-24T23:48:54.498604+0000","last_unstale":"2024-04-24T23:49:00.864612+0000","last_undegraded":"2024-04-24T23:49:00.864612+0000","last_fullsized":"2024-04-24T23:49:00.864612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:15:10.269984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526077+0000","last_change":"2024-04-24T23:48:59.524031+0000","last_active":"2024-04-24T23:49:00.526077+0000","last_peered":"2024-04-24T23:49:00.526077+0000","last_clean":"2024-04-24T23:49:00.526077+0000","last_became_active":"2024-04-24T23:48:59.523585+0000","last_became_peered":"2024-04-24T23:48:59.523585+0000","last_unstale":"2024-04-24T23:49:00.526077+0000","last_undegraded":"2024-04-24T23:49:00.526077+0000","last_fullsized":"2024-04-24T23:49:00.526077+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:33.409892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863757+0000","last_change":"2024-04-24T23:48:59.539246+0000","last_active":"2024-04-24T23:49:00.863757+0000","last_peered":"2024-04-24T23:49:00.863757+0000","last_clean":"2024-04-24T23:49:00.863757+0000","last_became_active":"2024-04-24T23:48:59.538845+0000","last_became_peered":"2024-04-24T23:48:59.538845+0000","last_unstale":"2024-04-24T23:49:00.863757+0000","last_undegraded":"2024-04-24T23:49:00.863757+0000","last_fullsized":"2024-04-24T23:49:00.863757+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:37.247560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508807+0000","last_change":"2024-04-24T23:48:55.493299+0000","last_active":"2024-04-24T23:49:00.508807+0000","last_peered":"2024-04-24T23:49:00.508807+0000","last_clean":"2024-04-24T23:49:00.508807+0000","last_became_active":"2024-04-24T23:48:55.492944+0000","last_became_peered":"2024-04-24T23:48:55.492944+0000","last_unstale":"2024-04-24T23:49:00.508807+0000","last_undegraded":"2024-04-24T23:49:00.508807+0000","last_fullsized":"2024-04-24T23:49:00.508807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:53:38.263114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863928+0000","last_change":"2024-04-24T23:48:54.494727+0000","last_active":"2024-04-24T23:49:00.863928+0000","last_peered":"2024-04-24T23:49:00.863928+0000","last_clean":"2024-04-24T23:49:00.863928+0000","last_became_active":"2024-04-24T23:48:54.494441+0000","last_became_peered":"2024-04-24T23:48:54.494441+0000","last_unstale":"2024-04-24T23:49:00.863928+0000","last_undegraded":"2024-04-24T23:49:00.863928+0000","last_fullsized":"2024-04-24T23:49:00.863928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864551+0000","last_change":"2024-04-24T23:48:59.530686+0000","last_active":"2024-04-24T23:49:00.864551+0000","last_peered":"2024-04-24T23:49:00.864551+0000","last_clean":"2024-04-24T23:49:00.864551+0000","last_became_active":"2024-04-24T23:48:59.529705+0000","last_became_peered":"2024-04-24T23:48:59.529705+0000","last_unstale":"2024-04-24T23:49:00.864551+0000","last_undegraded":"2024-04-24T23:49:00.864551+0000","last_fullsized":"2024-04-24T23:49:00.864551+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:10.292844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526143+0000","last_change":"2024-04-24T23:48:55.498040+0000","last_active":"2024-04-24T23:49:00.526143+0000","last_peered":"2024-04-24T23:49:00.526143+0000","last_clean":"2024-04-24T23:49:00.526143+0000","last_became_active":"2024-04-24T23:48:55.497298+0000","last_became_peered":"2024-04-24T23:48:55.497298+0000","last_unstale":"2024-04-24T23:49:00.526143+0000","last_undegraded":"2024-04-24T23:49:00.526143+0000","last_fullsized":"2024-04-24T23:49:00.526143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526688+0000","last_change":"2024-04-24T23:48:54.495330+0000","last_active":"2024-04-24T23:49:00.526688+0000","last_peered":"2024-04-24T23:49:00.526688+0000","last_clean":"2024-04-24T23:49:00.526688+0000","last_became_active":"2024-04-24T23:48:54.495079+0000","last_became_peered":"2024-04-24T23:48:54.495079+0000","last_unstale":"2024-04-24T23:49:00.526688+0000","last_undegraded":"2024-04-24T23:49:00.526688+0000","last_fullsized":"2024-04-24T23:49:00.526688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:16:25.658826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864192+0000","last_change":"2024-04-24T23:48:54.492734+0000","last_active":"2024-04-24T23:49:00.864192+0000","last_peered":"2024-04-24T23:49:00.864192+0000","last_clean":"2024-04-24T23:49:00.864192+0000","last_became_active":"2024-04-24T23:48:54.492467+0000","last_became_peered":"2024-04-24T23:48:54.492467+0000","last_unstale":"2024-04-24T23:49:00.864192+0000","last_undegraded":"2024-04-24T23:49:00.864192+0000","last_fullsized":"2024-04-24T23:49:00.864192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864956+0000","last_change":"2024-04-24T23:48:55.507847+0000","last_active":"2024-04-24T23:49:00.864956+0000","last_peered":"2024-04-24T23:49:00.864956+0000","last_clean":"2024-04-24T23:49:00.864956+0000","last_became_active":"2024-04-24T23:48:55.507382+0000","last_became_peered":"2024-04-24T23:48:55.507382+0000","last_unstale":"2024-04-24T23:49:00.864956+0000","last_undegraded":"2024-04-24T23:49:00.864956+0000","last_fullsized":"2024-04-24T23:49:00.864956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863934+0000","last_change":"2024-04-24T23:48:59.521454+0000","last_active":"2024-04-24T23:49:00.863934+0000","last_peered":"2024-04-24T23:49:00.863934+0000","last_clean":"2024-04-24T23:49:00.863934+0000","last_became_active":"2024-04-24T23:48:59.521041+0000","last_became_peered":"2024-04-24T23:48:59.521041+0000","last_unstale":"2024-04-24T23:49:00.863934+0000","last_undegraded":"2024-04-24T23:49:00.863934+0000","last_fullsized":"2024-04-24T23:49:00.863934+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:42.836048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526295+0000","last_change":"2024-04-24T23:48:54.489072+0000","last_active":"2024-04-24T23:49:00.526295+0000","last_peered":"2024-04-24T23:49:00.526295+0000","last_clean":"2024-04-24T23:49:00.526295+0000","last_became_active":"2024-04-24T23:48:54.488735+0000","last_became_peered":"2024-04-24T23:48:54.488735+0000","last_unstale":"2024-04-24T23:49:00.526295+0000","last_undegraded":"2024-04-24T23:49:00.526295+0000","last_fullsized":"2024-04-24T23:49:00.526295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:26.511856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510340+0000","last_change":"2024-04-24T23:48:55.501095+0000","last_active":"2024-04-24T23:49:00.510340+0000","last_peered":"2024-04-24T23:49:00.510340+0000","last_clean":"2024-04-24T23:49:00.510340+0000","last_became_active":"2024-04-24T23:48:55.500804+0000","last_became_peered":"2024-04-24T23:48:55.500804+0000","last_unstale":"2024-04-24T23:49:00.510340+0000","last_undegraded":"2024-04-24T23:49:00.510340+0000","last_fullsized":"2024-04-24T23:49:00.510340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:15:11.274374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526847+0000","last_change":"2024-04-24T23:48:59.532387+0000","last_active":"2024-04-24T23:49:00.526847+0000","last_peered":"2024-04-24T23:49:00.526847+0000","last_clean":"2024-04-24T23:49:00.526847+0000","last_became_active":"2024-04-24T23:48:59.531805+0000","last_became_peered":"2024-04-24T23:48:59.531805+0000","last_unstale":"2024-04-24T23:49:00.526847+0000","last_undegraded":"2024-04-24T23:49:00.526847+0000","last_fullsized":"2024-04-24T23:49:00.526847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:10.597258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863811+0000","last_change":"2024-04-24T23:48:55.502131+0000","last_active":"2024-04-24T23:49:00.863811+0000","last_peered":"2024-04-24T23:49:00.863811+0000","last_clean":"2024-04-24T23:49:00.863811+0000","last_became_active":"2024-04-24T23:48:55.501871+0000","last_became_peered":"2024-04-24T23:48:55.501871+0000","last_unstale":"2024-04-24T23:49:00.863811+0000","last_undegraded":"2024-04-24T23:49:00.863811+0000","last_fullsized":"2024-04-24T23:49:00.863811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863465+0000","last_change":"2024-04-24T23:48:55.503813+0000","last_active":"2024-04-24T23:49:00.863465+0000","last_peered":"2024-04-24T23:49:00.863465+0000","last_clean":"2024-04-24T23:49:00.863465+0000","last_became_active":"2024-04-24T23:48:55.503517+0000","last_became_peered":"2024-04-24T23:48:55.503517+0000","last_unstale":"2024-04-24T23:49:00.863465+0000","last_undegraded":"2024-04-24T23:49:00.863465+0000","last_fullsized":"2024-04-24T23:49:00.863465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:25.555824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864441+0000","last_change":"2024-04-24T23:48:55.507595+0000","last_active":"2024-04-24T23:49:00.864441+0000","last_peered":"2024-04-24T23:49:00.864441+0000","last_clean":"2024-04-24T23:49:00.864441+0000","last_became_active":"2024-04-24T23:48:55.507307+0000","last_became_peered":"2024-04-24T23:48:55.507307+0000","last_unstale":"2024-04-24T23:49:00.864441+0000","last_undegraded":"2024-04-24T23:49:00.864441+0000","last_fullsized":"2024-04-24T23:49:00.864441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:10.290840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864867+0000","last_change":"2024-04-24T23:48:55.498028+0000","last_active":"2024-04-24T23:49:00.864867+0000","last_peered":"2024-04-24T23:49:00.864867+0000","last_clean":"2024-04-24T23:49:00.864867+0000","last_became_active":"2024-04-24T23:48:55.496715+0000","last_became_peered":"2024-04-24T23:48:55.496715+0000","last_unstale":"2024-04-24T23:49:00.864867+0000","last_undegraded":"2024-04-24T23:49:00.864867+0000","last_fullsized":"2024-04-24T23:49:00.864867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526777+0000","last_change":"2024-04-24T23:48:55.505719+0000","last_active":"2024-04-24T23:49:00.526777+0000","last_peered":"2024-04-24T23:49:00.526777+0000","last_clean":"2024-04-24T23:49:00.526777+0000","last_became_active":"2024-04-24T23:48:55.505418+0000","last_became_peered":"2024-04-24T23:48:55.505418+0000","last_unstale":"2024-04-24T23:49:00.526777+0000","last_undegraded":"2024-04-24T23:49:00.526777+0000","last_fullsized":"2024-04-24T23:49:00.526777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509327+0000","last_change":"2024-04-24T23:48:55.492403+0000","last_active":"2024-04-24T23:49:00.509327+0000","last_peered":"2024-04-24T23:49:00.509327+0000","last_clean":"2024-04-24T23:49:00.509327+0000","last_became_active":"2024-04-24T23:48:55.492112+0000","last_became_peered":"2024-04-24T23:48:55.492112+0000","last_unstale":"2024-04-24T23:49:00.509327+0000","last_undegraded":"2024-04-24T23:49:00.509327+0000","last_fullsized":"2024-04-24T23:49:00.509327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864731+0000","last_change":"2024-04-24T23:48:55.488480+0000","last_active":"2024-04-24T23:49:00.864731+0000","last_peered":"2024-04-24T23:49:00.864731+0000","last_clean":"2024-04-24T23:49:00.864731+0000","last_became_active":"2024-04-24T23:48:55.488134+0000","last_became_peered":"2024-04-24T23:48:55.488134+0000","last_unstale":"2024-04-24T23:49:00.864731+0000","last_undegraded":"2024-04-24T23:49:00.864731+0000","last_fullsized":"2024-04-24T23:49:00.864731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:55.758743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526760+0000","last_change":"2024-04-24T23:48:55.507589+0000","last_active":"2024-04-24T23:49:00.526760+0000","last_peered":"2024-04-24T23:49:00.526760+0000","last_clean":"2024-04-24T23:49:00.526760+0000","last_became_active":"2024-04-24T23:48:55.507281+0000","last_became_peered":"2024-04-24T23:48:55.507281+0000","last_unstale":"2024-04-24T23:49:00.526760+0000","last_undegraded":"2024-04-24T23:49:00.526760+0000","last_fullsized":"2024-04-24T23:49:00.526760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:51.930595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863920+0000","last_change":"2024-04-24T23:48:55.498917+0000","last_active":"2024-04-24T23:49:00.863920+0000","last_peered":"2024-04-24T23:49:00.863920+0000","last_clean":"2024-04-24T23:49:00.863920+0000","last_became_active":"2024-04-24T23:48:55.498627+0000","last_became_peered":"2024-04-24T23:48:55.498627+0000","last_unstale":"2024-04-24T23:49:00.863920+0000","last_undegraded":"2024-04-24T23:49:00.863920+0000","last_fullsized":"2024-04-24T23:49:00.863920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:55.758743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526124+0000","last_change":"2024-04-24T23:48:55.507449+0000","last_active":"2024-04-24T23:49:00.526124+0000","last_peered":"2024-04-24T23:49:00.526124+0000","last_clean":"2024-04-24T23:49:00.526124+0000","last_became_active":"2024-04-24T23:48:55.507151+0000","last_became_peered":"2024-04-24T23:48:55.507151+0000","last_unstale":"2024-04-24T23:49:00.526124+0000","last_undegraded":"2024-04-24T23:49:00.526124+0000","last_fullsized":"2024-04-24T23:49:00.526124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:46.889872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864334+0000","last_change":"2024-04-24T23:48:55.511263+0000","last_active":"2024-04-24T23:49:00.864334+0000","last_peered":"2024-04-24T23:49:00.864334+0000","last_clean":"2024-04-24T23:49:00.864334+0000","last_became_active":"2024-04-24T23:48:55.510422+0000","last_became_peered":"2024-04-24T23:48:55.510422+0000","last_unstale":"2024-04-24T23:49:00.864334+0000","last_undegraded":"2024-04-24T23:49:00.864334+0000","last_fullsized":"2024-04-24T23:49:00.864334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:47.627183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864769+0000","last_change":"2024-04-24T23:48:55.508355+0000","last_active":"2024-04-24T23:49:00.864769+0000","last_peered":"2024-04-24T23:49:00.864769+0000","last_clean":"2024-04-24T23:49:00.864769+0000","last_became_active":"2024-04-24T23:48:55.506836+0000","last_became_peered":"2024-04-24T23:48:55.506836+0000","last_unstale":"2024-04-24T23:49:00.864769+0000","last_undegraded":"2024-04-24T23:49:00.864769+0000","last_fullsized":"2024-04-24T23:49:00.864769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:53:38.263114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864802+0000","last_change":"2024-04-24T23:48:55.507774+0000","last_active":"2024-04-24T23:49:00.864802+0000","last_peered":"2024-04-24T23:49:00.864802+0000","last_clean":"2024-04-24T23:49:00.864802+0000","last_became_active":"2024-04-24T23:48:55.506816+0000","last_became_peered":"2024-04-24T23:48:55.506816+0000","last_unstale":"2024-04-24T23:49:00.864802+0000","last_undegraded":"2024-04-24T23:49:00.864802+0000","last_fullsized":"2024-04-24T23:49:00.864802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510851+0000","last_change":"2024-04-24T23:48:55.492334+0000","last_active":"2024-04-24T23:49:00.510851+0000","last_peered":"2024-04-24T23:49:00.510851+0000","last_clean":"2024-04-24T23:49:00.510851+0000","last_became_active":"2024-04-24T23:48:55.492065+0000","last_became_peered":"2024-04-24T23:48:55.492065+0000","last_unstale":"2024-04-24T23:49:00.510851+0000","last_undegraded":"2024-04-24T23:49:00.510851+0000","last_fullsized":"2024-04-24T23:49:00.510851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:42.483054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863835+0000","last_change":"2024-04-24T23:48:55.495084+0000","last_active":"2024-04-24T23:49:00.863835+0000","last_peered":"2024-04-24T23:49:00.863835+0000","last_clean":"2024-04-24T23:49:00.863835+0000","last_became_active":"2024-04-24T23:48:55.494776+0000","last_became_peered":"2024-04-24T23:48:55.494776+0000","last_unstale":"2024-04-24T23:49:00.863835+0000","last_undegraded":"2024-04-24T23:49:00.863835+0000","last_fullsized":"2024-04-24T23:49:00.863835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:32.416739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865234+0000","last_change":"2024-04-24T23:48:55.511141+0000","last_active":"2024-04-24T23:49:00.865234+0000","last_peered":"2024-04-24T23:49:00.865234+0000","last_clean":"2024-04-24T23:49:00.865234+0000","last_became_active":"2024-04-24T23:48:55.510836+0000","last_became_peered":"2024-04-24T23:48:55.510836+0000","last_unstale":"2024-04-24T23:49:00.865234+0000","last_undegraded":"2024-04-24T23:49:00.865234+0000","last_fullsized":"2024-04-24T23:49:00.865234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:18.684092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863787+0000","last_change":"2024-04-24T23:48:55.508632+0000","last_active":"2024-04-24T23:49:00.863787+0000","last_peered":"2024-04-24T23:49:00.863787+0000","last_clean":"2024-04-24T23:49:00.863787+0000","last_became_active":"2024-04-24T23:48:55.507448+0000","last_became_peered":"2024-04-24T23:48:55.507448+0000","last_unstale":"2024-04-24T23:49:00.863787+0000","last_undegraded":"2024-04-24T23:49:00.863787+0000","last_fullsized":"2024-04-24T23:49:00.863787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:46.889872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508815+0000","last_change":"2024-04-24T23:48:55.508181+0000","last_active":"2024-04-24T23:49:00.508815+0000","last_peered":"2024-04-24T23:49:00.508815+0000","last_clean":"2024-04-24T23:49:00.508815+0000","last_became_active":"2024-04-24T23:48:55.507877+0000","last_became_peered":"2024-04-24T23:48:55.507877+0000","last_unstale":"2024-04-24T23:49:00.508815+0000","last_undegraded":"2024-04-24T23:49:00.508815+0000","last_fullsized":"2024-04-24T23:49:00.508815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864457+0000","last_change":"2024-04-24T23:48:55.497468+0000","last_active":"2024-04-24T23:49:00.864457+0000","last_peered":"2024-04-24T23:49:00.864457+0000","last_clean":"2024-04-24T23:49:00.864457+0000","last_became_active":"2024-04-24T23:48:55.496754+0000","last_became_peered":"2024-04-24T23:48:55.496754+0000","last_unstale":"2024-04-24T23:49:00.864457+0000","last_undegraded":"2024-04-24T23:49:00.864457+0000","last_fullsized":"2024-04-24T23:49:00.864457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:58.209422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526144+0000","last_change":"2024-04-24T23:48:55.492306+0000","last_active":"2024-04-24T23:49:00.526144+0000","last_peered":"2024-04-24T23:49:00.526144+0000","last_clean":"2024-04-24T23:49:00.526144+0000","last_became_active":"2024-04-24T23:48:55.492008+0000","last_became_peered":"2024-04-24T23:48:55.492008+0000","last_unstale":"2024-04-24T23:49:00.526144+0000","last_undegraded":"2024-04-24T23:49:00.526144+0000","last_fullsized":"2024-04-24T23:49:00.526144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:55.758743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510496+0000","last_change":"2024-04-24T23:48:55.501767+0000","last_active":"2024-04-24T23:49:00.510496+0000","last_peered":"2024-04-24T23:49:00.510496+0000","last_clean":"2024-04-24T23:49:00.510496+0000","last_became_active":"2024-04-24T23:48:55.500776+0000","last_became_peered":"2024-04-24T23:48:55.500776+0000","last_unstale":"2024-04-24T23:49:00.510496+0000","last_undegraded":"2024-04-24T23:49:00.510496+0000","last_fullsized":"2024-04-24T23:49:00.510496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:04.085761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864571+0000","last_change":"2024-04-24T23:48:55.511220+0000","last_active":"2024-04-24T23:49:00.864571+0000","last_peered":"2024-04-24T23:49:00.864571+0000","last_clean":"2024-04-24T23:49:00.864571+0000","last_became_active":"2024-04-24T23:48:55.510937+0000","last_became_peered":"2024-04-24T23:48:55.510937+0000","last_unstale":"2024-04-24T23:49:00.864571+0000","last_undegraded":"2024-04-24T23:49:00.864571+0000","last_fullsized":"2024-04-24T23:49:00.864571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864622+0000","last_change":"2024-04-24T23:48:55.503843+0000","last_active":"2024-04-24T23:49:00.864622+0000","last_peered":"2024-04-24T23:49:00.864622+0000","last_clean":"2024-04-24T23:49:00.864622+0000","last_became_active":"2024-04-24T23:48:55.503569+0000","last_became_peered":"2024-04-24T23:48:55.503569+0000","last_unstale":"2024-04-24T23:49:00.864622+0000","last_undegraded":"2024-04-24T23:49:00.864622+0000","last_fullsized":"2024-04-24T23:49:00.864622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:46.889872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511357+0000","last_change":"2024-04-24T23:48:55.509090+0000","last_active":"2024-04-24T23:49:00.511357+0000","last_peered":"2024-04-24T23:49:00.511357+0000","last_clean":"2024-04-24T23:49:00.511357+0000","last_became_active":"2024-04-24T23:48:55.508730+0000","last_became_peered":"2024-04-24T23:48:55.508730+0000","last_unstale":"2024-04-24T23:49:00.511357+0000","last_undegraded":"2024-04-24T23:49:00.511357+0000","last_fullsized":"2024-04-24T23:49:00.511357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:18.684092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.525876+0000","last_change":"2024-04-24T23:48:55.498351+0000","last_active":"2024-04-24T23:49:00.525876+0000","last_peered":"2024-04-24T23:49:00.525876+0000","last_clean":"2024-04-24T23:49:00.525876+0000","last_became_active":"2024-04-24T23:48:55.497694+0000","last_became_peered":"2024-04-24T23:48:55.497694+0000","last_unstale":"2024-04-24T23:49:00.525876+0000","last_undegraded":"2024-04-24T23:49:00.525876+0000","last_fullsized":"2024-04-24T23:49:00.525876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509595+0000","last_change":"2024-04-24T23:48:55.501677+0000","last_active":"2024-04-24T23:49:00.509595+0000","last_peered":"2024-04-24T23:49:00.509595+0000","last_clean":"2024-04-24T23:49:00.509595+0000","last_became_active":"2024-04-24T23:48:55.501359+0000","last_became_peered":"2024-04-24T23:48:55.501359+0000","last_unstale":"2024-04-24T23:49:00.509595+0000","last_undegraded":"2024-04-24T23:49:00.509595+0000","last_fullsized":"2024-04-24T23:49:00.509595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:47.627183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864613+0000","last_change":"2024-04-24T23:48:55.511089+0000","last_active":"2024-04-24T23:49:00.864613+0000","last_peered":"2024-04-24T23:49:00.864613+0000","last_clean":"2024-04-24T23:49:00.864613+0000","last_became_active":"2024-04-24T23:48:55.510818+0000","last_became_peered":"2024-04-24T23:48:55.510818+0000","last_unstale":"2024-04-24T23:49:00.864613+0000","last_undegraded":"2024-04-24T23:49:00.864613+0000","last_fullsized":"2024-04-24T23:49:00.864613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:03:28.472120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509524+0000","last_change":"2024-04-24T23:48:55.492973+0000","last_active":"2024-04-24T23:49:00.509524+0000","last_peered":"2024-04-24T23:49:00.509524+0000","last_clean":"2024-04-24T23:49:00.509524+0000","last_became_active":"2024-04-24T23:48:55.492713+0000","last_became_peered":"2024-04-24T23:48:55.492713+0000","last_unstale":"2024-04-24T23:49:00.509524+0000","last_undegraded":"2024-04-24T23:49:00.509524+0000","last_fullsized":"2024-04-24T23:49:00.509524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:10.290840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864561+0000","last_change":"2024-04-24T23:48:55.508975+0000","last_active":"2024-04-24T23:49:00.864561+0000","last_peered":"2024-04-24T23:49:00.864561+0000","last_clean":"2024-04-24T23:49:00.864561+0000","last_became_active":"2024-04-24T23:48:55.507970+0000","last_became_peered":"2024-04-24T23:48:55.507970+0000","last_unstale":"2024-04-24T23:49:00.864561+0000","last_undegraded":"2024-04-24T23:49:00.864561+0000","last_fullsized":"2024-04-24T23:49:00.864561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:10.235664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864614+0000","last_change":"2024-04-24T23:48:55.500274+0000","last_active":"2024-04-24T23:49:00.864614+0000","last_peered":"2024-04-24T23:49:00.864614+0000","last_clean":"2024-04-24T23:49:00.864614+0000","last_became_active":"2024-04-24T23:48:55.500004+0000","last_became_peered":"2024-04-24T23:48:55.500004+0000","last_unstale":"2024-04-24T23:49:00.864614+0000","last_undegraded":"2024-04-24T23:49:00.864614+0000","last_fullsized":"2024-04-24T23:49:00.864614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:02:47.989664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509710+0000","last_change":"2024-04-24T23:48:55.499555+0000","last_active":"2024-04-24T23:49:00.509710+0000","last_peered":"2024-04-24T23:49:00.509710+0000","last_clean":"2024-04-24T23:49:00.509710+0000","last_became_active":"2024-04-24T23:48:55.499163+0000","last_became_peered":"2024-04-24T23:48:55.499163+0000","last_unstale":"2024-04-24T23:49:00.509710+0000","last_undegraded":"2024-04-24T23:49:00.509710+0000","last_fullsized":"2024-04-24T23:49:00.509710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511249+0000","last_change":"2024-04-24T23:48:55.493200+0000","last_active":"2024-04-24T23:49:00.511249+0000","last_peered":"2024-04-24T23:49:00.511249+0000","last_clean":"2024-04-24T23:49:00.511249+0000","last_became_active":"2024-04-24T23:48:55.491655+0000","last_became_peered":"2024-04-24T23:48:55.491655+0000","last_unstale":"2024-04-24T23:49:00.511249+0000","last_undegraded":"2024-04-24T23:49:00.511249+0000","last_fullsized":"2024-04-24T23:49:00.511249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:04.801548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:49:19.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:49:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.694+0000 7f87b4653640 1 Processor -- start 2024-04-24T23:49:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.694+0000 7f87b4653640 1 -- start start 2024-04-24T23:49:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.695+0000 7f87b4653640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f87ac15bcd0 0x7f87ac1601f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.695+0000 7f87b4653640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87ac0d16c0 0x7f87ac06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.695+0000 7f87b4653640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87ac1607d0 con 0x7f87ac0705b0 2024-04-24T23:49:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.695+0000 7f87b4653640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f87ac06f660 con 0x7f87ac0d16c0 2024-04-24T23:49:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.695+0000 7f87b4653640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f87ac06ed30 con 0x7f87ac15bcd0 2024-04-24T23:49:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.695+0000 7f87b2bf0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87ac0d16c0 0x7f87ac06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.695+0000 7f87b2bf0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87ac0d16c0 0x7f87ac06faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48266/0 (socket says 172.21.15.16:48266) 2024-04-24T23:49:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.695+0000 7f87b2bf0640 1 -- 172.21.15.16:0/3561525248 learned_addr learned my addr 172.21.15.16:0/3561525248 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.695+0000 7f87b23ef640 1 --2- 172.21.15.16:0/3561525248 >> v2:172.21.15.156:3300/0 conn(0x7f87ac15bcd0 0x7f87ac1601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.695+0000 7f87b2bf0640 1 -- 172.21.15.16:0/3561525248 >> v2:172.21.15.156:3300/0 conn(0x7f87ac15bcd0 msgr2=0x7f87ac1601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.695+0000 7f87b2bf0640 1 --2- 172.21.15.16:0/3561525248 >> v2:172.21.15.156:3300/0 conn(0x7f87ac15bcd0 0x7f87ac1601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.695+0000 7f87b2bf0640 1 -- 172.21.15.16:0/3561525248 >> v1:172.21.15.156:6789/0 conn(0x7f87ac0705b0 legacy=0x7f87ac0d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:19.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.695+0000 7f87b2bf0640 1 -- 172.21.15.16:0/3561525248 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87ac06f010 con 0x7f87ac0d16c0 2024-04-24T23:49:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.695+0000 7f87b23ef640 1 --2- 172.21.15.16:0/3561525248 >> v2:172.21.15.156:3300/0 conn(0x7f87ac15bcd0 0x7f87ac1601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.696+0000 7f87b2bf0640 1 --2- 172.21.15.16:0/3561525248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0d16c0 0x7f87ac06faf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f87ac070580 tx=0x7f87a402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54177623503dad9e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.696+0000 7f87b13ed640 1 -- 172.21.15.16:0/3561525248 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a4036070 con 0x7f87ac0d16c0 2024-04-24T23:49:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.696+0000 7f87b13ed640 1 -- 172.21.15.16:0/3561525248 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87a4031600 con 0x7f87ac0d16c0 2024-04-24T23:49:19.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.696+0000 7f87b13ed640 1 -- 172.21.15.16:0/3561525248 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a40319d0 con 0x7f87ac0d16c0 2024-04-24T23:49:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.697+0000 7f87b4653640 1 -- 172.21.15.16:0/3561525248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0d16c0 msgr2=0x7f87ac06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.697+0000 7f87b4653640 1 --2- 172.21.15.16:0/3561525248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0d16c0 0x7f87ac06faf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f87ac070580 tx=0x7f87a402a5d0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.697+0000 7f87b4653640 1 -- 172.21.15.16:0/3561525248 shutdown_connections 2024-04-24T23:49:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.697+0000 7f87b4653640 1 --2- 172.21.15.16:0/3561525248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0d16c0 0x7f87ac06faf0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.697+0000 7f87b4653640 1 --2- 172.21.15.16:0/3561525248 >> v2:172.21.15.156:3300/0 conn(0x7f87ac15bcd0 0x7f87ac1601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.697+0000 7f87b4653640 1 -- 172.21.15.16:0/3561525248 >> 172.21.15.16:0/3561525248 conn(0x7f87ac06b4d0 msgr2=0x7f87ac0b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:19.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.697+0000 7f87b4653640 1 -- 172.21.15.16:0/3561525248 shutdown_connections 2024-04-24T23:49:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.697+0000 7f87b4653640 1 -- 172.21.15.16:0/3561525248 wait complete. 2024-04-24T23:49:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.698+0000 7f87b4653640 1 Processor -- start 2024-04-24T23:49:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.698+0000 7f87b4653640 1 -- start start 2024-04-24T23:49:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b4653640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87ac0705b0 0x7f87ac0b6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b4653640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0d16c0 0x7f87ac151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b4653640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87ac15bcd0 0x7f87ac15e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b4653640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac0d09d0 con 0x7f87ac0d16c0 2024-04-24T23:49:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b4653640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac15c0b0 con 0x7f87ac0705b0 2024-04-24T23:49:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b4653640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f87ac0d0b50 con 0x7f87ac15bcd0 2024-04-24T23:49:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b1bee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0d16c0 0x7f87ac151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b1bee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0d16c0 0x7f87ac151630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48268/0 (socket says 172.21.15.16:48268) 2024-04-24T23:49:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b1bee640 1 -- 172.21.15.16:0/1070229250 learned_addr learned my addr 172.21.15.16:0/1070229250 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b23ef640 1 --2- 172.21.15.16:0/1070229250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87ac0705b0 0x7f87ac0b6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b1bee640 1 -- 172.21.15.16:0/1070229250 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87ac15bcd0 msgr2=0x7f87ac15e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b1bee640 1 --2- 172.21.15.16:0/1070229250 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87ac15bcd0 0x7f87ac15e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b1bee640 1 -- 172.21.15.16:0/1070229250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87ac0705b0 msgr2=0x7f87ac0b6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b1bee640 1 --2- 172.21.15.16:0/1070229250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87ac0705b0 0x7f87ac0b6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b1bee640 1 -- 172.21.15.16:0/1070229250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87ac0b7080 con 0x7f87ac0d16c0 2024-04-24T23:49:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87b1bee640 1 --2- 172.21.15.16:0/1070229250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0d16c0 0x7f87ac151630 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f879c00b3f0 tx=0x7f879c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87a2ffd640 1 -- 172.21.15.16:0/1070229250 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c015020 con 0x7f87ac0d16c0 2024-04-24T23:49:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.699+0000 7f87a2ffd640 1 -- 172.21.15.16:0/1070229250 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f879c011830 con 0x7f87ac0d16c0 2024-04-24T23:49:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.700+0000 7f87b4653640 1 -- 172.21.15.16:0/1070229250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87ac0b5de0 con 0x7f87ac0d16c0 2024-04-24T23:49:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.700+0000 7f87a2ffd640 1 -- 172.21.15.16:0/1070229250 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f879c011c20 con 0x7f87ac0d16c0 2024-04-24T23:49:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.700+0000 7f87b4653640 1 -- 172.21.15.16:0/1070229250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87ac0b6620 con 0x7f87ac0d16c0 2024-04-24T23:49:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.701+0000 7f87b4653640 1 -- 172.21.15.16:0/1070229250 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87ac0b6820 con 0x7f87ac0d16c0 2024-04-24T23:49:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.701+0000 7f87a2ffd640 1 -- 172.21.15.16:0/1070229250 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f879c018020 con 0x7f87ac0d16c0 2024-04-24T23:49:19.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.701+0000 7f87a2ffd640 1 --2- 172.21.15.16:0/1070229250 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f879006be30 0x7f879006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.701+0000 7f87a2ffd640 1 -- 172.21.15.16:0/1070229250 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f879c08b100 con 0x7f87ac0d16c0 2024-04-24T23:49:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.701+0000 7f87b23ef640 1 --2- 172.21.15.16:0/1070229250 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f879006be30 0x7f879006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:19.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.702+0000 7f87b23ef640 1 --2- 172.21.15.16:0/1070229250 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f879006be30 0x7f879006e2f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0d0390 tx=0x7f87a8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:19.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.706+0000 7f87a2ffd640 1 -- 172.21.15.16:0/1070229250 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f87ac0b6820 con 0x7f87ac0d16c0 2024-04-24T23:49:19.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.884+0000 7f87b4653640 1 -- 172.21.15.16:0/1070229250 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f87ac0b5fc0 con 0x7f879006be30 2024-04-24T23:49:19.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.892+0000 7f87a2ffd640 1 -- 172.21.15.16:0/1070229250 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332534 (secure 0 0 0) 0x7f87a800bdb0 con 0x7f879006be30 2024-04-24T23:49:19.897 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:19.899 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-24T23:49:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.895+0000 7f87b4653640 1 -- 172.21.15.16:0/1070229250 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f879006be30 msgr2=0x7f879006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.895+0000 7f87b4653640 1 --2- 172.21.15.16:0/1070229250 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f879006be30 0x7f879006e2f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f87ac0d0390 tx=0x7f87a8008040 comp rx=0 tx=0).stop 2024-04-24T23:49:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.895+0000 7f87b4653640 1 -- 172.21.15.16:0/1070229250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0d16c0 msgr2=0x7f87ac151630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.895+0000 7f87b4653640 1 --2- 172.21.15.16:0/1070229250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0d16c0 0x7f87ac151630 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f879c00b3f0 tx=0x7f879c00b8b0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.895+0000 7f87b4653640 1 -- 172.21.15.16:0/1070229250 shutdown_connections 2024-04-24T23:49:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.895+0000 7f87b4653640 1 --2- 172.21.15.16:0/1070229250 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f879006be30 0x7f879006e2f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.895+0000 7f87b4653640 1 --2- 172.21.15.16:0/1070229250 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f87ac15bcd0 0x7f87ac15e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.895+0000 7f87b4653640 1 --2- 172.21.15.16:0/1070229250 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87ac0d16c0 0x7f87ac151630 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.895+0000 7f87b4653640 1 --2- 172.21.15.16:0/1070229250 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87ac0705b0 0x7f87ac0b6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:19.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.895+0000 7f87b4653640 1 -- 172.21.15.16:0/1070229250 >> 172.21.15.16:0/1070229250 conn(0x7f87ac06b4d0 msgr2=0x7f87ac0cd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.895+0000 7f87b4653640 1 -- 172.21.15.16:0/1070229250 shutdown_connections 2024-04-24T23:49:19.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:19.895+0000 7f87b4653640 1 -- 172.21.15.16:0/1070229250 wait complete. 2024-04-24T23:49:19.906 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-24T23:49:19.122640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248620,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754726100,"statfs":{"total":773094113280,"available":772839526400,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2230,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003107"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864269+0000","last_change":"2024-04-24T23:48:54.496506+0000","last_active":"2024-04-24T23:49:00.864269+0000","last_peered":"2024-04-24T23:49:00.864269+0000","last_clean":"2024-04-24T23:49:00.864269+0000","last_became_active":"2024-04-24T23:48:54.496184+0000","last_became_peered":"2024-04-24T23:48:54.496184+0000","last_unstale":"2024-04-24T23:49:00.864269+0000","last_undegraded":"2024-04-24T23:49:00.864269+0000","last_fullsized":"2024-04-24T23:49:00.864269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:26.511856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526502+0000","last_change":"2024-04-24T23:48:55.493918+0000","last_active":"2024-04-24T23:49:00.526502+0000","last_peered":"2024-04-24T23:49:00.526502+0000","last_clean":"2024-04-24T23:49:00.526502+0000","last_became_active":"2024-04-24T23:48:55.493534+0000","last_became_peered":"2024-04-24T23:48:55.493534+0000","last_unstale":"2024-04-24T23:49:00.526502+0000","last_undegraded":"2024-04-24T23:49:00.526502+0000","last_fullsized":"2024-04-24T23:49:00.526502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:56:24.335063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.525963+0000","last_change":"2024-04-24T23:48:59.535521+0000","last_active":"2024-04-24T23:49:00.525963+0000","last_peered":"2024-04-24T23:49:00.525963+0000","last_clean":"2024-04-24T23:49:00.525963+0000","last_became_active":"2024-04-24T23:48:59.535120+0000","last_became_peered":"2024-04-24T23:48:59.535120+0000","last_unstale":"2024-04-24T23:49:00.525963+0000","last_undegraded":"2024-04-24T23:49:00.525963+0000","last_fullsized":"2024-04-24T23:49:00.525963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:02:34.201356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526615+0000","last_change":"2024-04-24T23:48:54.494612+0000","last_active":"2024-04-24T23:49:00.526615+0000","last_peered":"2024-04-24T23:49:00.526615+0000","last_clean":"2024-04-24T23:49:00.526615+0000","last_became_active":"2024-04-24T23:48:54.494317+0000","last_became_peered":"2024-04-24T23:48:54.494317+0000","last_unstale":"2024-04-24T23:49:00.526615+0000","last_undegraded":"2024-04-24T23:49:00.526615+0000","last_fullsized":"2024-04-24T23:49:00.526615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:07.349136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864094+0000","last_change":"2024-04-24T23:48:55.512152+0000","last_active":"2024-04-24T23:49:00.864094+0000","last_peered":"2024-04-24T23:49:00.864094+0000","last_clean":"2024-04-24T23:49:00.864094+0000","last_became_active":"2024-04-24T23:48:55.511396+0000","last_became_peered":"2024-04-24T23:48:55.511396+0000","last_unstale":"2024-04-24T23:49:00.864094+0000","last_undegraded":"2024-04-24T23:49:00.864094+0000","last_fullsized":"2024-04-24T23:49:00.864094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510457+0000","last_change":"2024-04-24T23:48:59.530584+0000","last_active":"2024-04-24T23:49:00.510457+0000","last_peered":"2024-04-24T23:49:00.510457+0000","last_clean":"2024-04-24T23:49:00.510457+0000","last_became_active":"2024-04-24T23:48:59.530081+0000","last_became_peered":"2024-04-24T23:48:59.530081+0000","last_unstale":"2024-04-24T23:49:00.510457+0000","last_undegraded":"2024-04-24T23:49:00.510457+0000","last_fullsized":"2024-04-24T23:49:00.510457+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:03:32.497072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510851+0000","last_change":"2024-04-24T23:48:54.495374+0000","last_active":"2024-04-24T23:49:00.510851+0000","last_peered":"2024-04-24T23:49:00.510851+0000","last_clean":"2024-04-24T23:49:00.510851+0000","last_became_active":"2024-04-24T23:48:54.495040+0000","last_became_peered":"2024-04-24T23:48:54.495040+0000","last_unstale":"2024-04-24T23:49:00.510851+0000","last_undegraded":"2024-04-24T23:49:00.510851+0000","last_fullsized":"2024-04-24T23:49:00.510851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:31.412349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864681+0000","last_change":"2024-04-24T23:48:55.505099+0000","last_active":"2024-04-24T23:49:00.864681+0000","last_peered":"2024-04-24T23:49:00.864681+0000","last_clean":"2024-04-24T23:49:00.864681+0000","last_became_active":"2024-04-24T23:48:55.504778+0000","last_became_peered":"2024-04-24T23:48:55.504778+0000","last_unstale":"2024-04-24T23:49:00.864681+0000","last_undegraded":"2024-04-24T23:49:00.864681+0000","last_fullsized":"2024-04-24T23:49:00.864681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:17.488901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511291+0000","last_change":"2024-04-24T23:48:59.526537+0000","last_active":"2024-04-24T23:49:00.511291+0000","last_peered":"2024-04-24T23:49:00.511291+0000","last_clean":"2024-04-24T23:49:00.511291+0000","last_became_active":"2024-04-24T23:48:59.526038+0000","last_became_peered":"2024-04-24T23:48:59.526038+0000","last_unstale":"2024-04-24T23:49:00.511291+0000","last_undegraded":"2024-04-24T23:49:00.511291+0000","last_fullsized":"2024-04-24T23:49:00.511291+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510029+0000","last_change":"2024-04-24T23:48:54.498311+0000","last_active":"2024-04-24T23:49:00.510029+0000","last_peered":"2024-04-24T23:49:00.510029+0000","last_clean":"2024-04-24T23:49:00.510029+0000","last_became_active":"2024-04-24T23:48:54.497986+0000","last_became_peered":"2024-04-24T23:48:54.497986+0000","last_unstale":"2024-04-24T23:49:00.510029+0000","last_undegraded":"2024-04-24T23:49:00.510029+0000","last_fullsized":"2024-04-24T23:49:00.510029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865047+0000","last_change":"2024-04-24T23:48:55.507029+0000","last_active":"2024-04-24T23:49:00.865047+0000","last_peered":"2024-04-24T23:49:00.865047+0000","last_clean":"2024-04-24T23:49:00.865047+0000","last_became_active":"2024-04-24T23:48:55.506772+0000","last_became_peered":"2024-04-24T23:48:55.506772+0000","last_unstale":"2024-04-24T23:49:00.865047+0000","last_undegraded":"2024-04-24T23:49:00.865047+0000","last_fullsized":"2024-04-24T23:49:00.865047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:42.483054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509920+0000","last_change":"2024-04-24T23:48:59.530333+0000","last_active":"2024-04-24T23:49:00.509920+0000","last_peered":"2024-04-24T23:49:00.509920+0000","last_clean":"2024-04-24T23:49:00.509920+0000","last_became_active":"2024-04-24T23:48:59.529831+0000","last_became_peered":"2024-04-24T23:48:59.529831+0000","last_unstale":"2024-04-24T23:49:00.509920+0000","last_undegraded":"2024-04-24T23:49:00.509920+0000","last_fullsized":"2024-04-24T23:49:00.509920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:47.852102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864057+0000","last_change":"2024-04-24T23:48:55.504704+0000","last_active":"2024-04-24T23:49:00.864057+0000","last_peered":"2024-04-24T23:49:00.864057+0000","last_clean":"2024-04-24T23:49:00.864057+0000","last_became_active":"2024-04-24T23:48:55.504444+0000","last_became_peered":"2024-04-24T23:48:55.504444+0000","last_unstale":"2024-04-24T23:49:00.864057+0000","last_undegraded":"2024-04-24T23:49:00.864057+0000","last_fullsized":"2024-04-24T23:49:00.864057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510280+0000","last_change":"2024-04-24T23:48:54.493270+0000","last_active":"2024-04-24T23:49:00.510280+0000","last_peered":"2024-04-24T23:49:00.510280+0000","last_clean":"2024-04-24T23:49:00.510280+0000","last_became_active":"2024-04-24T23:48:54.492952+0000","last_became_peered":"2024-04-24T23:48:54.492952+0000","last_unstale":"2024-04-24T23:49:00.510280+0000","last_undegraded":"2024-04-24T23:49:00.510280+0000","last_fullsized":"2024-04-24T23:49:00.510280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:36:57.277988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510165+0000","last_change":"2024-04-24T23:48:59.530237+0000","last_active":"2024-04-24T23:49:00.510165+0000","last_peered":"2024-04-24T23:49:00.510165+0000","last_clean":"2024-04-24T23:49:00.510165+0000","last_became_active":"2024-04-24T23:48:59.529732+0000","last_became_peered":"2024-04-24T23:48:59.529732+0000","last_unstale":"2024-04-24T23:49:00.510165+0000","last_undegraded":"2024-04-24T23:49:00.510165+0000","last_fullsized":"2024-04-24T23:49:00.510165+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:59.783695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526172+0000","last_change":"2024-04-24T23:48:55.500061+0000","last_active":"2024-04-24T23:49:00.526172+0000","last_peered":"2024-04-24T23:49:00.526172+0000","last_clean":"2024-04-24T23:49:00.526172+0000","last_became_active":"2024-04-24T23:48:55.499515+0000","last_became_peered":"2024-04-24T23:48:55.499515+0000","last_unstale":"2024-04-24T23:49:00.526172+0000","last_undegraded":"2024-04-24T23:49:00.526172+0000","last_fullsized":"2024-04-24T23:49:00.526172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:10.235664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509506+0000","last_change":"2024-04-24T23:48:54.496643+0000","last_active":"2024-04-24T23:49:00.509506+0000","last_peered":"2024-04-24T23:49:00.509506+0000","last_clean":"2024-04-24T23:49:00.509506+0000","last_became_active":"2024-04-24T23:48:54.496367+0000","last_became_peered":"2024-04-24T23:48:54.496367+0000","last_unstale":"2024-04-24T23:49:00.509506+0000","last_undegraded":"2024-04-24T23:49:00.509506+0000","last_fullsized":"2024-04-24T23:49:00.509506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:17.375124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510348+0000","last_change":"2024-04-24T23:48:59.531327+0000","last_active":"2024-04-24T23:49:00.510348+0000","last_peered":"2024-04-24T23:49:00.510348+0000","last_clean":"2024-04-24T23:49:00.510348+0000","last_became_active":"2024-04-24T23:48:59.530959+0000","last_became_peered":"2024-04-24T23:48:59.530959+0000","last_unstale":"2024-04-24T23:49:00.510348+0000","last_undegraded":"2024-04-24T23:49:00.510348+0000","last_fullsized":"2024-04-24T23:49:00.510348+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:14.260616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526263+0000","last_change":"2024-04-24T23:48:55.494442+0000","last_active":"2024-04-24T23:49:00.526263+0000","last_peered":"2024-04-24T23:49:00.526263+0000","last_clean":"2024-04-24T23:49:00.526263+0000","last_became_active":"2024-04-24T23:48:55.494165+0000","last_became_peered":"2024-04-24T23:48:55.494165+0000","last_unstale":"2024-04-24T23:49:00.526263+0000","last_undegraded":"2024-04-24T23:49:00.526263+0000","last_fullsized":"2024-04-24T23:49:00.526263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508952+0000","last_change":"2024-04-24T23:48:54.488761+0000","last_active":"2024-04-24T23:49:00.508952+0000","last_peered":"2024-04-24T23:49:00.508952+0000","last_clean":"2024-04-24T23:49:00.508952+0000","last_became_active":"2024-04-24T23:48:54.488431+0000","last_became_peered":"2024-04-24T23:48:54.488431+0000","last_unstale":"2024-04-24T23:49:00.508952+0000","last_undegraded":"2024-04-24T23:49:00.508952+0000","last_fullsized":"2024-04-24T23:49:00.508952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509720+0000","last_change":"2024-04-24T23:48:59.523973+0000","last_active":"2024-04-24T23:49:00.509720+0000","last_peered":"2024-04-24T23:49:00.509720+0000","last_clean":"2024-04-24T23:49:00.509720+0000","last_became_active":"2024-04-24T23:48:59.520679+0000","last_became_peered":"2024-04-24T23:48:59.520679+0000","last_unstale":"2024-04-24T23:49:00.509720+0000","last_undegraded":"2024-04-24T23:49:00.509720+0000","last_fullsized":"2024-04-24T23:49:00.509720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511072+0000","last_change":"2024-04-24T23:48:55.505618+0000","last_active":"2024-04-24T23:49:00.511072+0000","last_peered":"2024-04-24T23:49:00.511072+0000","last_clean":"2024-04-24T23:49:00.511072+0000","last_became_active":"2024-04-24T23:48:55.504804+0000","last_became_peered":"2024-04-24T23:48:55.504804+0000","last_unstale":"2024-04-24T23:49:00.511072+0000","last_undegraded":"2024-04-24T23:49:00.511072+0000","last_fullsized":"2024-04-24T23:49:00.511072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:46.215368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509409+0000","last_change":"2024-04-24T23:48:54.496898+0000","last_active":"2024-04-24T23:49:00.509409+0000","last_peered":"2024-04-24T23:49:00.509409+0000","last_clean":"2024-04-24T23:49:00.509409+0000","last_became_active":"2024-04-24T23:48:54.496603+0000","last_became_peered":"2024-04-24T23:48:54.496603+0000","last_unstale":"2024-04-24T23:49:00.509409+0000","last_undegraded":"2024-04-24T23:49:00.509409+0000","last_fullsized":"2024-04-24T23:49:00.509409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510695+0000","last_change":"2024-04-24T23:48:59.533868+0000","last_active":"2024-04-24T23:49:00.510695+0000","last_peered":"2024-04-24T23:49:00.510695+0000","last_clean":"2024-04-24T23:49:00.510695+0000","last_became_active":"2024-04-24T23:48:59.533487+0000","last_became_peered":"2024-04-24T23:48:59.533487+0000","last_unstale":"2024-04-24T23:49:00.510695+0000","last_undegraded":"2024-04-24T23:49:00.510695+0000","last_fullsized":"2024-04-24T23:49:00.510695+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:21.513853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864071+0000","last_change":"2024-04-24T23:48:55.497734+0000","last_active":"2024-04-24T23:49:00.864071+0000","last_peered":"2024-04-24T23:49:00.864071+0000","last_clean":"2024-04-24T23:49:00.864071+0000","last_became_active":"2024-04-24T23:48:55.496634+0000","last_became_peered":"2024-04-24T23:48:55.496634+0000","last_unstale":"2024-04-24T23:49:00.864071+0000","last_undegraded":"2024-04-24T23:49:00.864071+0000","last_fullsized":"2024-04-24T23:49:00.864071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:56:24.335063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864834+0000","last_change":"2024-04-24T23:48:54.499625+0000","last_active":"2024-04-24T23:49:00.864834+0000","last_peered":"2024-04-24T23:49:00.864834+0000","last_clean":"2024-04-24T23:49:00.864834+0000","last_became_active":"2024-04-24T23:48:54.499357+0000","last_became_peered":"2024-04-24T23:48:54.499357+0000","last_unstale":"2024-04-24T23:49:00.864834+0000","last_undegraded":"2024-04-24T23:49:00.864834+0000","last_fullsized":"2024-04-24T23:49:00.864834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511165+0000","last_change":"2024-04-24T23:48:59.523817+0000","last_active":"2024-04-24T23:49:00.511165+0000","last_peered":"2024-04-24T23:49:00.511165+0000","last_clean":"2024-04-24T23:49:00.511165+0000","last_became_active":"2024-04-24T23:48:59.522505+0000","last_became_peered":"2024-04-24T23:48:59.522505+0000","last_unstale":"2024-04-24T23:49:00.511165+0000","last_undegraded":"2024-04-24T23:49:00.511165+0000","last_fullsized":"2024-04-24T23:49:00.511165+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:37.247560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526836+0000","last_change":"2024-04-24T23:48:55.490605+0000","last_active":"2024-04-24T23:49:00.526836+0000","last_peered":"2024-04-24T23:49:00.526836+0000","last_clean":"2024-04-24T23:49:00.526836+0000","last_became_active":"2024-04-24T23:48:55.490271+0000","last_became_peered":"2024-04-24T23:48:55.490271+0000","last_unstale":"2024-04-24T23:49:00.526836+0000","last_undegraded":"2024-04-24T23:49:00.526836+0000","last_fullsized":"2024-04-24T23:49:00.526836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:53:38.263114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509803+0000","last_change":"2024-04-24T23:48:54.493101+0000","last_active":"2024-04-24T23:49:00.509803+0000","last_peered":"2024-04-24T23:49:00.509803+0000","last_clean":"2024-04-24T23:49:00.509803+0000","last_became_active":"2024-04-24T23:48:54.492768+0000","last_became_peered":"2024-04-24T23:48:54.492768+0000","last_unstale":"2024-04-24T23:49:00.509803+0000","last_undegraded":"2024-04-24T23:49:00.509803+0000","last_fullsized":"2024-04-24T23:49:00.509803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:15:10.269984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.525949+0000","last_change":"2024-04-24T23:48:59.535686+0000","last_active":"2024-04-24T23:49:00.525949+0000","last_peered":"2024-04-24T23:49:00.525949+0000","last_clean":"2024-04-24T23:49:00.525949+0000","last_became_active":"2024-04-24T23:48:59.535320+0000","last_became_peered":"2024-04-24T23:48:59.535320+0000","last_unstale":"2024-04-24T23:49:00.525949+0000","last_undegraded":"2024-04-24T23:49:00.525949+0000","last_fullsized":"2024-04-24T23:49:00.525949+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:22.709044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509618+0000","last_change":"2024-04-24T23:48:55.493935+0000","last_active":"2024-04-24T23:49:00.509618+0000","last_peered":"2024-04-24T23:49:00.509618+0000","last_clean":"2024-04-24T23:49:00.509618+0000","last_became_active":"2024-04-24T23:48:55.493604+0000","last_became_peered":"2024-04-24T23:48:55.493604+0000","last_unstale":"2024-04-24T23:49:00.509618+0000","last_undegraded":"2024-04-24T23:49:00.509618+0000","last_fullsized":"2024-04-24T23:49:00.509618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510971+0000","last_change":"2024-04-24T23:48:54.497633+0000","last_active":"2024-04-24T23:49:00.510971+0000","last_peered":"2024-04-24T23:49:00.510971+0000","last_clean":"2024-04-24T23:49:00.510971+0000","last_became_active":"2024-04-24T23:48:54.497329+0000","last_became_peered":"2024-04-24T23:48:54.497329+0000","last_unstale":"2024-04-24T23:49:00.510971+0000","last_undegraded":"2024-04-24T23:49:00.510971+0000","last_fullsized":"2024-04-24T23:49:00.510971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:37.806706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864823+0000","last_change":"2024-04-24T23:48:59.538720+0000","last_active":"2024-04-24T23:49:00.864823+0000","last_peered":"2024-04-24T23:49:00.864823+0000","last_clean":"2024-04-24T23:49:00.864823+0000","last_became_active":"2024-04-24T23:48:59.538310+0000","last_became_peered":"2024-04-24T23:48:59.538310+0000","last_unstale":"2024-04-24T23:49:00.864823+0000","last_undegraded":"2024-04-24T23:49:00.864823+0000","last_fullsized":"2024-04-24T23:49:00.864823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:10.292844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865110+0000","last_change":"2024-04-24T23:48:55.510066+0000","last_active":"2024-04-24T23:49:00.865110+0000","last_peered":"2024-04-24T23:49:00.865110+0000","last_clean":"2024-04-24T23:49:00.865110+0000","last_became_active":"2024-04-24T23:48:55.507977+0000","last_became_peered":"2024-04-24T23:48:55.507977+0000","last_unstale":"2024-04-24T23:49:00.865110+0000","last_undegraded":"2024-04-24T23:49:00.865110+0000","last_fullsized":"2024-04-24T23:49:00.865110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:06.267893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509767+0000","last_change":"2024-04-24T23:48:54.493032+0000","last_active":"2024-04-24T23:49:00.509767+0000","last_peered":"2024-04-24T23:49:00.509767+0000","last_clean":"2024-04-24T23:49:00.509767+0000","last_became_active":"2024-04-24T23:48:54.492694+0000","last_became_peered":"2024-04-24T23:48:54.492694+0000","last_unstale":"2024-04-24T23:49:00.509767+0000","last_undegraded":"2024-04-24T23:49:00.509767+0000","last_fullsized":"2024-04-24T23:49:00.509767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509213+0000","last_change":"2024-04-24T23:48:59.527964+0000","last_active":"2024-04-24T23:49:00.509213+0000","last_peered":"2024-04-24T23:49:00.509213+0000","last_clean":"2024-04-24T23:49:00.509213+0000","last_became_active":"2024-04-24T23:48:59.527519+0000","last_became_peered":"2024-04-24T23:48:59.527519+0000","last_unstale":"2024-04-24T23:49:00.509213+0000","last_undegraded":"2024-04-24T23:49:00.509213+0000","last_fullsized":"2024-04-24T23:49:00.509213+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:37.247560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511053+0000","last_change":"2024-04-24T23:48:55.505703+0000","last_active":"2024-04-24T23:49:00.511053+0000","last_peered":"2024-04-24T23:49:00.511053+0000","last_clean":"2024-04-24T23:49:00.511053+0000","last_became_active":"2024-04-24T23:48:55.505257+0000","last_became_peered":"2024-04-24T23:48:55.505257+0000","last_unstale":"2024-04-24T23:49:00.511053+0000","last_undegraded":"2024-04-24T23:49:00.511053+0000","last_fullsized":"2024-04-24T23:49:00.511053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:41.372867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864734+0000","last_change":"2024-04-24T23:48:54.494961+0000","last_active":"2024-04-24T23:49:00.864734+0000","last_peered":"2024-04-24T23:49:00.864734+0000","last_clean":"2024-04-24T23:49:00.864734+0000","last_became_active":"2024-04-24T23:48:54.494672+0000","last_became_peered":"2024-04-24T23:48:54.494672+0000","last_unstale":"2024-04-24T23:49:00.864734+0000","last_undegraded":"2024-04-24T23:49:00.864734+0000","last_fullsized":"2024-04-24T23:49:00.864734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509770+0000","last_change":"2024-04-24T23:48:59.525981+0000","last_active":"2024-04-24T23:49:00.509770+0000","last_peered":"2024-04-24T23:49:00.509770+0000","last_clean":"2024-04-24T23:49:00.509770+0000","last_became_active":"2024-04-24T23:48:59.524788+0000","last_became_peered":"2024-04-24T23:48:59.524788+0000","last_unstale":"2024-04-24T23:49:00.509770+0000","last_undegraded":"2024-04-24T23:49:00.509770+0000","last_fullsized":"2024-04-24T23:49:00.509770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:08.826500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508934+0000","last_change":"2024-04-24T23:48:55.493966+0000","last_active":"2024-04-24T23:49:00.508934+0000","last_peered":"2024-04-24T23:49:00.508934+0000","last_clean":"2024-04-24T23:49:00.508934+0000","last_became_active":"2024-04-24T23:48:55.493622+0000","last_became_peered":"2024-04-24T23:48:55.493622+0000","last_unstale":"2024-04-24T23:49:00.508934+0000","last_undegraded":"2024-04-24T23:49:00.508934+0000","last_fullsized":"2024-04-24T23:49:00.508934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863565+0000","last_change":"2024-04-24T23:48:54.497811+0000","last_active":"2024-04-24T23:49:00.863565+0000","last_peered":"2024-04-24T23:49:00.863565+0000","last_clean":"2024-04-24T23:49:00.863565+0000","last_became_active":"2024-04-24T23:48:54.497465+0000","last_became_peered":"2024-04-24T23:48:54.497465+0000","last_unstale":"2024-04-24T23:49:00.863565+0000","last_undegraded":"2024-04-24T23:49:00.863565+0000","last_fullsized":"2024-04-24T23:49:00.863565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:17.375124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509953+0000","last_change":"2024-04-24T23:48:59.536596+0000","last_active":"2024-04-24T23:49:00.509953+0000","last_peered":"2024-04-24T23:49:00.509953+0000","last_clean":"2024-04-24T23:49:00.509953+0000","last_became_active":"2024-04-24T23:48:59.536186+0000","last_became_peered":"2024-04-24T23:48:59.536186+0000","last_unstale":"2024-04-24T23:49:00.509953+0000","last_undegraded":"2024-04-24T23:49:00.509953+0000","last_fullsized":"2024-04-24T23:49:00.509953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:46.508006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863695+0000","last_change":"2024-04-24T23:48:55.507490+0000","last_active":"2024-04-24T23:49:00.863695+0000","last_peered":"2024-04-24T23:49:00.863695+0000","last_clean":"2024-04-24T23:49:00.863695+0000","last_became_active":"2024-04-24T23:48:55.506415+0000","last_became_peered":"2024-04-24T23:48:55.506415+0000","last_unstale":"2024-04-24T23:49:00.863695+0000","last_undegraded":"2024-04-24T23:49:00.863695+0000","last_fullsized":"2024-04-24T23:49:00.863695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509313+0000","last_change":"2024-04-24T23:48:54.491388+0000","last_active":"2024-04-24T23:49:00.509313+0000","last_peered":"2024-04-24T23:49:00.509313+0000","last_clean":"2024-04-24T23:49:00.509313+0000","last_became_active":"2024-04-24T23:48:54.490991+0000","last_became_peered":"2024-04-24T23:48:54.490991+0000","last_unstale":"2024-04-24T23:49:00.509313+0000","last_undegraded":"2024-04-24T23:49:00.509313+0000","last_fullsized":"2024-04-24T23:49:00.509313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863944+0000","last_change":"2024-04-24T23:48:59.539922+0000","last_active":"2024-04-24T23:49:00.863944+0000","last_peered":"2024-04-24T23:49:00.863944+0000","last_clean":"2024-04-24T23:49:00.863944+0000","last_became_active":"2024-04-24T23:48:59.539449+0000","last_became_peered":"2024-04-24T23:48:59.539449+0000","last_unstale":"2024-04-24T23:49:00.863944+0000","last_undegraded":"2024-04-24T23:49:00.863944+0000","last_fullsized":"2024-04-24T23:49:00.863944+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:08.826500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526078+0000","last_change":"2024-04-24T23:48:55.509237+0000","last_active":"2024-04-24T23:49:00.526078+0000","last_peered":"2024-04-24T23:49:00.526078+0000","last_clean":"2024-04-24T23:49:00.526078+0000","last_became_active":"2024-04-24T23:48:55.508847+0000","last_became_peered":"2024-04-24T23:48:55.508847+0000","last_unstale":"2024-04-24T23:49:00.526078+0000","last_undegraded":"2024-04-24T23:49:00.526078+0000","last_fullsized":"2024-04-24T23:49:00.526078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864955+0000","last_change":"2024-04-24T23:48:54.496310+0000","last_active":"2024-04-24T23:49:00.864955+0000","last_peered":"2024-04-24T23:49:00.864955+0000","last_clean":"2024-04-24T23:49:00.864955+0000","last_became_active":"2024-04-24T23:48:54.496016+0000","last_became_peered":"2024-04-24T23:48:54.496016+0000","last_unstale":"2024-04-24T23:49:00.864955+0000","last_undegraded":"2024-04-24T23:49:00.864955+0000","last_fullsized":"2024-04-24T23:49:00.864955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509522+0000","last_change":"2024-04-24T23:48:45.082459+0000","last_active":"2024-04-24T23:49:00.509522+0000","last_peered":"2024-04-24T23:49:00.509522+0000","last_clean":"2024-04-24T23:49:00.509522+0000","last_became_active":"2024-04-24T23:48:45.081495+0000","last_became_peered":"2024-04-24T23:48:45.081495+0000","last_unstale":"2024-04-24T23:49:00.509522+0000","last_undegraded":"2024-04-24T23:49:00.509522+0000","last_fullsized":"2024-04-24T23:49:00.509522+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_clean_scrub_stamp":"2024-04-24T23:48:43.440993+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:53:20.832511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864431+0000","last_change":"2024-04-24T23:48:59.521532+0000","last_active":"2024-04-24T23:49:00.864431+0000","last_peered":"2024-04-24T23:49:00.864431+0000","last_clean":"2024-04-24T23:49:00.864431+0000","last_became_active":"2024-04-24T23:48:59.521144+0000","last_became_peered":"2024-04-24T23:48:59.521144+0000","last_unstale":"2024-04-24T23:49:00.864431+0000","last_undegraded":"2024-04-24T23:49:00.864431+0000","last_fullsized":"2024-04-24T23:49:00.864431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:08.826500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510601+0000","last_change":"2024-04-24T23:48:55.509123+0000","last_active":"2024-04-24T23:49:00.510601+0000","last_peered":"2024-04-24T23:49:00.510601+0000","last_clean":"2024-04-24T23:49:00.510601+0000","last_became_active":"2024-04-24T23:48:55.508860+0000","last_became_peered":"2024-04-24T23:48:55.508860+0000","last_unstale":"2024-04-24T23:49:00.510601+0000","last_undegraded":"2024-04-24T23:49:00.510601+0000","last_fullsized":"2024-04-24T23:49:00.510601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:03:28.472120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863884+0000","last_change":"2024-04-24T23:48:54.498741+0000","last_active":"2024-04-24T23:49:00.863884+0000","last_peered":"2024-04-24T23:49:00.863884+0000","last_clean":"2024-04-24T23:49:00.863884+0000","last_became_active":"2024-04-24T23:48:54.498241+0000","last_became_peered":"2024-04-24T23:48:54.498241+0000","last_unstale":"2024-04-24T23:49:00.863884+0000","last_undegraded":"2024-04-24T23:49:00.863884+0000","last_fullsized":"2024-04-24T23:49:00.863884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:07.349136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864020+0000","last_change":"2024-04-24T23:48:59.528434+0000","last_active":"2024-04-24T23:49:00.864020+0000","last_peered":"2024-04-24T23:49:00.864020+0000","last_clean":"2024-04-24T23:49:00.864020+0000","last_became_active":"2024-04-24T23:48:59.528062+0000","last_became_peered":"2024-04-24T23:48:59.528062+0000","last_unstale":"2024-04-24T23:49:00.864020+0000","last_undegraded":"2024-04-24T23:49:00.864020+0000","last_fullsized":"2024-04-24T23:49:00.864020+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864801+0000","last_change":"2024-04-24T23:48:55.509799+0000","last_active":"2024-04-24T23:49:00.864801+0000","last_peered":"2024-04-24T23:49:00.864801+0000","last_clean":"2024-04-24T23:49:00.864801+0000","last_became_active":"2024-04-24T23:48:55.509286+0000","last_became_peered":"2024-04-24T23:48:55.509286+0000","last_unstale":"2024-04-24T23:49:00.864801+0000","last_undegraded":"2024-04-24T23:49:00.864801+0000","last_fullsized":"2024-04-24T23:49:00.864801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:33.222608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509751+0000","last_change":"2024-04-24T23:48:54.490614+0000","last_active":"2024-04-24T23:49:00.509751+0000","last_peered":"2024-04-24T23:49:00.509751+0000","last_clean":"2024-04-24T23:49:00.509751+0000","last_became_active":"2024-04-24T23:48:54.490029+0000","last_became_peered":"2024-04-24T23:48:54.490029+0000","last_unstale":"2024-04-24T23:49:00.509751+0000","last_undegraded":"2024-04-24T23:49:00.509751+0000","last_fullsized":"2024-04-24T23:49:00.509751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864962+0000","last_change":"2024-04-24T23:48:59.529966+0000","last_active":"2024-04-24T23:49:00.864962+0000","last_peered":"2024-04-24T23:49:00.864962+0000","last_clean":"2024-04-24T23:49:00.864962+0000","last_became_active":"2024-04-24T23:48:59.529508+0000","last_became_peered":"2024-04-24T23:48:59.529508+0000","last_unstale":"2024-04-24T23:49:00.864962+0000","last_undegraded":"2024-04-24T23:49:00.864962+0000","last_fullsized":"2024-04-24T23:49:00.864962+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:46.508006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511361+0000","last_change":"2024-04-24T23:48:55.506681+0000","last_active":"2024-04-24T23:49:00.511361+0000","last_peered":"2024-04-24T23:49:00.511361+0000","last_clean":"2024-04-24T23:49:00.511361+0000","last_became_active":"2024-04-24T23:48:55.506393+0000","last_became_peered":"2024-04-24T23:48:55.506393+0000","last_unstale":"2024-04-24T23:49:00.511361+0000","last_undegraded":"2024-04-24T23:49:00.511361+0000","last_fullsized":"2024-04-24T23:49:00.511361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:02:30.176405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863990+0000","last_change":"2024-04-24T23:48:54.499321+0000","last_active":"2024-04-24T23:49:00.863990+0000","last_peered":"2024-04-24T23:49:00.863990+0000","last_clean":"2024-04-24T23:49:00.863990+0000","last_became_active":"2024-04-24T23:48:54.499015+0000","last_became_peered":"2024-04-24T23:48:54.499015+0000","last_unstale":"2024-04-24T23:49:00.863990+0000","last_undegraded":"2024-04-24T23:49:00.863990+0000","last_fullsized":"2024-04-24T23:49:00.863990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863856+0000","last_change":"2024-04-24T23:48:59.522405+0000","last_active":"2024-04-24T23:49:00.863856+0000","last_peered":"2024-04-24T23:49:00.863856+0000","last_clean":"2024-04-24T23:49:00.863856+0000","last_became_active":"2024-04-24T23:48:59.521434+0000","last_became_peered":"2024-04-24T23:48:59.521434+0000","last_unstale":"2024-04-24T23:49:00.863856+0000","last_undegraded":"2024-04-24T23:49:00.863856+0000","last_fullsized":"2024-04-24T23:49:00.863856+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:39.512436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511297+0000","last_change":"2024-04-24T23:48:55.507123+0000","last_active":"2024-04-24T23:49:00.511297+0000","last_peered":"2024-04-24T23:49:00.511297+0000","last_clean":"2024-04-24T23:49:00.511297+0000","last_became_active":"2024-04-24T23:48:55.506838+0000","last_became_peered":"2024-04-24T23:48:55.506838+0000","last_unstale":"2024-04-24T23:49:00.511297+0000","last_undegraded":"2024-04-24T23:49:00.511297+0000","last_fullsized":"2024-04-24T23:49:00.511297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:17.488901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509852+0000","last_change":"2024-04-24T23:48:54.494375+0000","last_active":"2024-04-24T23:49:00.509852+0000","last_peered":"2024-04-24T23:49:00.509852+0000","last_clean":"2024-04-24T23:49:00.509852+0000","last_became_active":"2024-04-24T23:48:54.494080+0000","last_became_peered":"2024-04-24T23:48:54.494080+0000","last_unstale":"2024-04-24T23:49:00.509852+0000","last_undegraded":"2024-04-24T23:49:00.509852+0000","last_fullsized":"2024-04-24T23:49:00.509852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:46.268530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510042+0000","last_change":"2024-04-24T23:48:59.530805+0000","last_active":"2024-04-24T23:49:00.510042+0000","last_peered":"2024-04-24T23:49:00.510042+0000","last_clean":"2024-04-24T23:49:00.510042+0000","last_became_active":"2024-04-24T23:48:59.530126+0000","last_became_peered":"2024-04-24T23:48:59.530126+0000","last_unstale":"2024-04-24T23:49:00.510042+0000","last_undegraded":"2024-04-24T23:49:00.510042+0000","last_fullsized":"2024-04-24T23:49:00.510042+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:50.914824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509597+0000","last_change":"2024-04-24T23:48:55.499881+0000","last_active":"2024-04-24T23:49:00.509597+0000","last_peered":"2024-04-24T23:49:00.509597+0000","last_clean":"2024-04-24T23:49:00.509597+0000","last_became_active":"2024-04-24T23:48:55.499493+0000","last_became_peered":"2024-04-24T23:48:55.499493+0000","last_unstale":"2024-04-24T23:49:00.509597+0000","last_undegraded":"2024-04-24T23:49:00.509597+0000","last_fullsized":"2024-04-24T23:49:00.509597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:33.222608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863280+0000","last_change":"2024-04-24T23:48:54.493205+0000","last_active":"2024-04-24T23:49:00.863280+0000","last_peered":"2024-04-24T23:49:00.863280+0000","last_clean":"2024-04-24T23:49:00.863280+0000","last_became_active":"2024-04-24T23:48:54.492446+0000","last_became_peered":"2024-04-24T23:48:54.492446+0000","last_unstale":"2024-04-24T23:49:00.863280+0000","last_undegraded":"2024-04-24T23:49:00.863280+0000","last_fullsized":"2024-04-24T23:49:00.863280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:16:25.658826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863935+0000","last_change":"2024-04-24T23:48:59.532383+0000","last_active":"2024-04-24T23:49:00.863935+0000","last_peered":"2024-04-24T23:49:00.863935+0000","last_clean":"2024-04-24T23:49:00.863935+0000","last_became_active":"2024-04-24T23:48:59.529606+0000","last_became_peered":"2024-04-24T23:48:59.529606+0000","last_unstale":"2024-04-24T23:49:00.863935+0000","last_undegraded":"2024-04-24T23:49:00.863935+0000","last_fullsized":"2024-04-24T23:49:00.863935+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864681+0000","last_change":"2024-04-24T23:48:55.499374+0000","last_active":"2024-04-24T23:49:00.864681+0000","last_peered":"2024-04-24T23:49:00.864681+0000","last_clean":"2024-04-24T23:49:00.864681+0000","last_became_active":"2024-04-24T23:48:55.499107+0000","last_became_peered":"2024-04-24T23:48:55.499107+0000","last_unstale":"2024-04-24T23:49:00.864681+0000","last_undegraded":"2024-04-24T23:49:00.864681+0000","last_fullsized":"2024-04-24T23:49:00.864681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864217+0000","last_change":"2024-04-24T23:48:54.491505+0000","last_active":"2024-04-24T23:49:00.864217+0000","last_peered":"2024-04-24T23:49:00.864217+0000","last_clean":"2024-04-24T23:49:00.864217+0000","last_became_active":"2024-04-24T23:48:54.491230+0000","last_became_peered":"2024-04-24T23:48:54.491230+0000","last_unstale":"2024-04-24T23:49:00.864217+0000","last_undegraded":"2024-04-24T23:49:00.864217+0000","last_fullsized":"2024-04-24T23:49:00.864217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526800+0000","last_change":"2024-04-24T23:48:59.531330+0000","last_active":"2024-04-24T23:49:00.526800+0000","last_peered":"2024-04-24T23:49:00.526800+0000","last_clean":"2024-04-24T23:49:00.526800+0000","last_became_active":"2024-04-24T23:48:59.530916+0000","last_became_peered":"2024-04-24T23:48:59.530916+0000","last_unstale":"2024-04-24T23:49:00.526800+0000","last_undegraded":"2024-04-24T23:49:00.526800+0000","last_fullsized":"2024-04-24T23:49:00.526800+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:43.024671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510174+0000","last_change":"2024-04-24T23:48:55.502184+0000","last_active":"2024-04-24T23:49:00.510174+0000","last_peered":"2024-04-24T23:49:00.510174+0000","last_clean":"2024-04-24T23:49:00.510174+0000","last_became_active":"2024-04-24T23:48:55.501682+0000","last_became_peered":"2024-04-24T23:48:55.501682+0000","last_unstale":"2024-04-24T23:49:00.510174+0000","last_undegraded":"2024-04-24T23:49:00.510174+0000","last_fullsized":"2024-04-24T23:49:00.510174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:06.267893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864548+0000","last_change":"2024-04-24T23:48:54.499017+0000","last_active":"2024-04-24T23:49:00.864548+0000","last_peered":"2024-04-24T23:49:00.864548+0000","last_clean":"2024-04-24T23:49:00.864548+0000","last_became_active":"2024-04-24T23:48:54.498747+0000","last_became_peered":"2024-04-24T23:48:54.498747+0000","last_unstale":"2024-04-24T23:49:00.864548+0000","last_undegraded":"2024-04-24T23:49:00.864548+0000","last_fullsized":"2024-04-24T23:49:00.864548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526541+0000","last_change":"2024-04-24T23:48:59.529890+0000","last_active":"2024-04-24T23:49:00.526541+0000","last_peered":"2024-04-24T23:49:00.526541+0000","last_clean":"2024-04-24T23:49:00.526541+0000","last_became_active":"2024-04-24T23:48:59.529534+0000","last_became_peered":"2024-04-24T23:48:59.529534+0000","last_unstale":"2024-04-24T23:49:00.526541+0000","last_undegraded":"2024-04-24T23:49:00.526541+0000","last_fullsized":"2024-04-24T23:49:00.526541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:38:25.945723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864543+0000","last_change":"2024-04-24T23:48:55.497917+0000","last_active":"2024-04-24T23:49:00.864543+0000","last_peered":"2024-04-24T23:49:00.864543+0000","last_clean":"2024-04-24T23:49:00.864543+0000","last_became_active":"2024-04-24T23:48:55.497158+0000","last_became_peered":"2024-04-24T23:48:55.497158+0000","last_unstale":"2024-04-24T23:49:00.864543+0000","last_undegraded":"2024-04-24T23:49:00.864543+0000","last_fullsized":"2024-04-24T23:49:00.864543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:51.930595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865010+0000","last_change":"2024-04-24T23:48:54.491006+0000","last_active":"2024-04-24T23:49:00.865010+0000","last_peered":"2024-04-24T23:49:00.865010+0000","last_clean":"2024-04-24T23:49:00.865010+0000","last_became_active":"2024-04-24T23:48:54.489661+0000","last_became_peered":"2024-04-24T23:48:54.489661+0000","last_unstale":"2024-04-24T23:49:00.865010+0000","last_undegraded":"2024-04-24T23:49:00.865010+0000","last_fullsized":"2024-04-24T23:49:00.865010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:16:25.658826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864966+0000","last_change":"2024-04-24T23:48:59.521225+0000","last_active":"2024-04-24T23:49:00.864966+0000","last_peered":"2024-04-24T23:49:00.864966+0000","last_clean":"2024-04-24T23:49:00.864966+0000","last_became_active":"2024-04-24T23:48:59.520761+0000","last_became_peered":"2024-04-24T23:48:59.520761+0000","last_unstale":"2024-04-24T23:49:00.864966+0000","last_undegraded":"2024-04-24T23:49:00.864966+0000","last_fullsized":"2024-04-24T23:49:00.864966+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:38:25.945723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510500+0000","last_change":"2024-04-24T23:48:55.487564+0000","last_active":"2024-04-24T23:49:00.510500+0000","last_peered":"2024-04-24T23:49:00.510500+0000","last_clean":"2024-04-24T23:49:00.510500+0000","last_became_active":"2024-04-24T23:48:55.487272+0000","last_became_peered":"2024-04-24T23:48:55.487272+0000","last_unstale":"2024-04-24T23:49:00.510500+0000","last_undegraded":"2024-04-24T23:49:00.510500+0000","last_fullsized":"2024-04-24T23:49:00.510500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:10.235664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:04.927990+0000","last_change":"2024-04-24T23:48:54.500818+0000","last_active":"2024-04-24T23:49:04.927990+0000","last_peered":"2024-04-24T23:49:04.927990+0000","last_clean":"2024-04-24T23:49:04.927990+0000","last_became_active":"2024-04-24T23:48:54.500560+0000","last_became_peered":"2024-04-24T23:48:54.500560+0000","last_unstale":"2024-04-24T23:49:04.927990+0000","last_undegraded":"2024-04-24T23:49:04.927990+0000","last_fullsized":"2024-04-24T23:49:04.927990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509881+0000","last_change":"2024-04-24T23:48:59.530292+0000","last_active":"2024-04-24T23:49:00.509881+0000","last_peered":"2024-04-24T23:49:00.509881+0000","last_clean":"2024-04-24T23:49:00.509881+0000","last_became_active":"2024-04-24T23:48:59.529429+0000","last_became_peered":"2024-04-24T23:48:59.529429+0000","last_unstale":"2024-04-24T23:49:00.509881+0000","last_undegraded":"2024-04-24T23:49:00.509881+0000","last_fullsized":"2024-04-24T23:49:00.509881+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:10.292844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510600+0000","last_change":"2024-04-24T23:48:55.502480+0000","last_active":"2024-04-24T23:49:00.510600+0000","last_peered":"2024-04-24T23:49:00.510600+0000","last_clean":"2024-04-24T23:49:00.510600+0000","last_became_active":"2024-04-24T23:48:55.502104+0000","last_became_peered":"2024-04-24T23:48:55.502104+0000","last_unstale":"2024-04-24T23:49:00.510600+0000","last_undegraded":"2024-04-24T23:49:00.510600+0000","last_fullsized":"2024-04-24T23:49:00.510600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:25.555824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509534+0000","last_change":"2024-04-24T23:48:54.495473+0000","last_active":"2024-04-24T23:49:00.509534+0000","last_peered":"2024-04-24T23:49:00.509534+0000","last_clean":"2024-04-24T23:49:00.509534+0000","last_became_active":"2024-04-24T23:48:54.495173+0000","last_became_peered":"2024-04-24T23:48:54.495173+0000","last_unstale":"2024-04-24T23:49:00.509534+0000","last_undegraded":"2024-04-24T23:49:00.509534+0000","last_fullsized":"2024-04-24T23:49:00.509534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510914+0000","last_change":"2024-04-24T23:48:59.532057+0000","last_active":"2024-04-24T23:49:00.510914+0000","last_peered":"2024-04-24T23:49:00.510914+0000","last_clean":"2024-04-24T23:49:00.510914+0000","last_became_active":"2024-04-24T23:48:59.530378+0000","last_became_peered":"2024-04-24T23:48:59.530378+0000","last_unstale":"2024-04-24T23:49:00.510914+0000","last_undegraded":"2024-04-24T23:49:00.510914+0000","last_fullsized":"2024-04-24T23:49:00.510914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:51.652134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864215+0000","last_change":"2024-04-24T23:48:55.508829+0000","last_active":"2024-04-24T23:49:00.864215+0000","last_peered":"2024-04-24T23:49:00.864215+0000","last_clean":"2024-04-24T23:49:00.864215+0000","last_became_active":"2024-04-24T23:48:55.508042+0000","last_became_peered":"2024-04-24T23:48:55.508042+0000","last_unstale":"2024-04-24T23:49:00.864215+0000","last_undegraded":"2024-04-24T23:49:00.864215+0000","last_fullsized":"2024-04-24T23:49:00.864215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865051+0000","last_change":"2024-04-24T23:48:54.497886+0000","last_active":"2024-04-24T23:49:00.865051+0000","last_peered":"2024-04-24T23:49:00.865051+0000","last_clean":"2024-04-24T23:49:00.865051+0000","last_became_active":"2024-04-24T23:48:54.497630+0000","last_became_peered":"2024-04-24T23:48:54.497630+0000","last_unstale":"2024-04-24T23:49:00.865051+0000","last_undegraded":"2024-04-24T23:49:00.865051+0000","last_fullsized":"2024-04-24T23:49:00.865051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:31.412349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526184+0000","last_change":"2024-04-24T23:48:59.532080+0000","last_active":"2024-04-24T23:49:00.526184+0000","last_peered":"2024-04-24T23:49:00.526184+0000","last_clean":"2024-04-24T23:49:00.526184+0000","last_became_active":"2024-04-24T23:48:59.531298+0000","last_became_peered":"2024-04-24T23:48:59.531298+0000","last_unstale":"2024-04-24T23:49:00.526184+0000","last_undegraded":"2024-04-24T23:49:00.526184+0000","last_fullsized":"2024-04-24T23:49:00.526184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:39.512436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864909+0000","last_change":"2024-04-24T23:48:55.513313+0000","last_active":"2024-04-24T23:49:00.864909+0000","last_peered":"2024-04-24T23:49:00.864909+0000","last_clean":"2024-04-24T23:49:00.864909+0000","last_became_active":"2024-04-24T23:48:55.513021+0000","last_became_peered":"2024-04-24T23:48:55.513021+0000","last_unstale":"2024-04-24T23:49:00.864909+0000","last_undegraded":"2024-04-24T23:49:00.864909+0000","last_fullsized":"2024-04-24T23:49:00.864909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:04.801548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864612+0000","last_change":"2024-04-24T23:48:54.498900+0000","last_active":"2024-04-24T23:49:00.864612+0000","last_peered":"2024-04-24T23:49:00.864612+0000","last_clean":"2024-04-24T23:49:00.864612+0000","last_became_active":"2024-04-24T23:48:54.498604+0000","last_became_peered":"2024-04-24T23:48:54.498604+0000","last_unstale":"2024-04-24T23:49:00.864612+0000","last_undegraded":"2024-04-24T23:49:00.864612+0000","last_fullsized":"2024-04-24T23:49:00.864612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:15:10.269984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526077+0000","last_change":"2024-04-24T23:48:59.524031+0000","last_active":"2024-04-24T23:49:00.526077+0000","last_peered":"2024-04-24T23:49:00.526077+0000","last_clean":"2024-04-24T23:49:00.526077+0000","last_became_active":"2024-04-24T23:48:59.523585+0000","last_became_peered":"2024-04-24T23:48:59.523585+0000","last_unstale":"2024-04-24T23:49:00.526077+0000","last_undegraded":"2024-04-24T23:49:00.526077+0000","last_fullsized":"2024-04-24T23:49:00.526077+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:33.409892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863757+0000","last_change":"2024-04-24T23:48:59.539246+0000","last_active":"2024-04-24T23:49:00.863757+0000","last_peered":"2024-04-24T23:49:00.863757+0000","last_clean":"2024-04-24T23:49:00.863757+0000","last_became_active":"2024-04-24T23:48:59.538845+0000","last_became_peered":"2024-04-24T23:48:59.538845+0000","last_unstale":"2024-04-24T23:49:00.863757+0000","last_undegraded":"2024-04-24T23:49:00.863757+0000","last_fullsized":"2024-04-24T23:49:00.863757+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:37.247560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508807+0000","last_change":"2024-04-24T23:48:55.493299+0000","last_active":"2024-04-24T23:49:00.508807+0000","last_peered":"2024-04-24T23:49:00.508807+0000","last_clean":"2024-04-24T23:49:00.508807+0000","last_became_active":"2024-04-24T23:48:55.492944+0000","last_became_peered":"2024-04-24T23:48:55.492944+0000","last_unstale":"2024-04-24T23:49:00.508807+0000","last_undegraded":"2024-04-24T23:49:00.508807+0000","last_fullsized":"2024-04-24T23:49:00.508807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:53:38.263114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863928+0000","last_change":"2024-04-24T23:48:54.494727+0000","last_active":"2024-04-24T23:49:00.863928+0000","last_peered":"2024-04-24T23:49:00.863928+0000","last_clean":"2024-04-24T23:49:00.863928+0000","last_became_active":"2024-04-24T23:48:54.494441+0000","last_became_peered":"2024-04-24T23:48:54.494441+0000","last_unstale":"2024-04-24T23:49:00.863928+0000","last_undegraded":"2024-04-24T23:49:00.863928+0000","last_fullsized":"2024-04-24T23:49:00.863928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864551+0000","last_change":"2024-04-24T23:48:59.530686+0000","last_active":"2024-04-24T23:49:00.864551+0000","last_peered":"2024-04-24T23:49:00.864551+0000","last_clean":"2024-04-24T23:49:00.864551+0000","last_became_active":"2024-04-24T23:48:59.529705+0000","last_became_peered":"2024-04-24T23:48:59.529705+0000","last_unstale":"2024-04-24T23:49:00.864551+0000","last_undegraded":"2024-04-24T23:49:00.864551+0000","last_fullsized":"2024-04-24T23:49:00.864551+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:10.292844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526143+0000","last_change":"2024-04-24T23:48:55.498040+0000","last_active":"2024-04-24T23:49:00.526143+0000","last_peered":"2024-04-24T23:49:00.526143+0000","last_clean":"2024-04-24T23:49:00.526143+0000","last_became_active":"2024-04-24T23:48:55.497298+0000","last_became_peered":"2024-04-24T23:48:55.497298+0000","last_unstale":"2024-04-24T23:49:00.526143+0000","last_undegraded":"2024-04-24T23:49:00.526143+0000","last_fullsized":"2024-04-24T23:49:00.526143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526688+0000","last_change":"2024-04-24T23:48:54.495330+0000","last_active":"2024-04-24T23:49:00.526688+0000","last_peered":"2024-04-24T23:49:00.526688+0000","last_clean":"2024-04-24T23:49:00.526688+0000","last_became_active":"2024-04-24T23:48:54.495079+0000","last_became_peered":"2024-04-24T23:48:54.495079+0000","last_unstale":"2024-04-24T23:49:00.526688+0000","last_undegraded":"2024-04-24T23:49:00.526688+0000","last_fullsized":"2024-04-24T23:49:00.526688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:16:25.658826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864192+0000","last_change":"2024-04-24T23:48:54.492734+0000","last_active":"2024-04-24T23:49:00.864192+0000","last_peered":"2024-04-24T23:49:00.864192+0000","last_clean":"2024-04-24T23:49:00.864192+0000","last_became_active":"2024-04-24T23:48:54.492467+0000","last_became_peered":"2024-04-24T23:48:54.492467+0000","last_unstale":"2024-04-24T23:49:00.864192+0000","last_undegraded":"2024-04-24T23:49:00.864192+0000","last_fullsized":"2024-04-24T23:49:00.864192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864956+0000","last_change":"2024-04-24T23:48:55.507847+0000","last_active":"2024-04-24T23:49:00.864956+0000","last_peered":"2024-04-24T23:49:00.864956+0000","last_clean":"2024-04-24T23:49:00.864956+0000","last_became_active":"2024-04-24T23:48:55.507382+0000","last_became_peered":"2024-04-24T23:48:55.507382+0000","last_unstale":"2024-04-24T23:49:00.864956+0000","last_undegraded":"2024-04-24T23:49:00.864956+0000","last_fullsized":"2024-04-24T23:49:00.864956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863934+0000","last_change":"2024-04-24T23:48:59.521454+0000","last_active":"2024-04-24T23:49:00.863934+0000","last_peered":"2024-04-24T23:49:00.863934+0000","last_clean":"2024-04-24T23:49:00.863934+0000","last_became_active":"2024-04-24T23:48:59.521041+0000","last_became_peered":"2024-04-24T23:48:59.521041+0000","last_unstale":"2024-04-24T23:49:00.863934+0000","last_undegraded":"2024-04-24T23:49:00.863934+0000","last_fullsized":"2024-04-24T23:49:00.863934+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:42.836048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526295+0000","last_change":"2024-04-24T23:48:54.489072+0000","last_active":"2024-04-24T23:49:00.526295+0000","last_peered":"2024-04-24T23:49:00.526295+0000","last_clean":"2024-04-24T23:49:00.526295+0000","last_became_active":"2024-04-24T23:48:54.488735+0000","last_became_peered":"2024-04-24T23:48:54.488735+0000","last_unstale":"2024-04-24T23:49:00.526295+0000","last_undegraded":"2024-04-24T23:49:00.526295+0000","last_fullsized":"2024-04-24T23:49:00.526295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:26.511856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510340+0000","last_change":"2024-04-24T23:48:55.501095+0000","last_active":"2024-04-24T23:49:00.510340+0000","last_peered":"2024-04-24T23:49:00.510340+0000","last_clean":"2024-04-24T23:49:00.510340+0000","last_became_active":"2024-04-24T23:48:55.500804+0000","last_became_peered":"2024-04-24T23:48:55.500804+0000","last_unstale":"2024-04-24T23:49:00.510340+0000","last_undegraded":"2024-04-24T23:49:00.510340+0000","last_fullsized":"2024-04-24T23:49:00.510340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:15:11.274374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526847+0000","last_change":"2024-04-24T23:48:59.532387+0000","last_active":"2024-04-24T23:49:00.526847+0000","last_peered":"2024-04-24T23:49:00.526847+0000","last_clean":"2024-04-24T23:49:00.526847+0000","last_became_active":"2024-04-24T23:48:59.531805+0000","last_became_peered":"2024-04-24T23:48:59.531805+0000","last_unstale":"2024-04-24T23:49:00.526847+0000","last_undegraded":"2024-04-24T23:49:00.526847+0000","last_fullsized":"2024-04-24T23:49:00.526847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:10.597258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863811+0000","last_change":"2024-04-24T23:48:55.502131+0000","last_active":"2024-04-24T23:49:00.863811+0000","last_peered":"2024-04-24T23:49:00.863811+0000","last_clean":"2024-04-24T23:49:00.863811+0000","last_became_active":"2024-04-24T23:48:55.501871+0000","last_became_peered":"2024-04-24T23:48:55.501871+0000","last_unstale":"2024-04-24T23:49:00.863811+0000","last_undegraded":"2024-04-24T23:49:00.863811+0000","last_fullsized":"2024-04-24T23:49:00.863811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863465+0000","last_change":"2024-04-24T23:48:55.503813+0000","last_active":"2024-04-24T23:49:00.863465+0000","last_peered":"2024-04-24T23:49:00.863465+0000","last_clean":"2024-04-24T23:49:00.863465+0000","last_became_active":"2024-04-24T23:48:55.503517+0000","last_became_peered":"2024-04-24T23:48:55.503517+0000","last_unstale":"2024-04-24T23:49:00.863465+0000","last_undegraded":"2024-04-24T23:49:00.863465+0000","last_fullsized":"2024-04-24T23:49:00.863465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:25.555824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864441+0000","last_change":"2024-04-24T23:48:55.507595+0000","last_active":"2024-04-24T23:49:00.864441+0000","last_peered":"2024-04-24T23:49:00.864441+0000","last_clean":"2024-04-24T23:49:00.864441+0000","last_became_active":"2024-04-24T23:48:55.507307+0000","last_became_peered":"2024-04-24T23:48:55.507307+0000","last_unstale":"2024-04-24T23:49:00.864441+0000","last_undegraded":"2024-04-24T23:49:00.864441+0000","last_fullsized":"2024-04-24T23:49:00.864441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:10.290840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864867+0000","last_change":"2024-04-24T23:48:55.498028+0000","last_active":"2024-04-24T23:49:00.864867+0000","last_peered":"2024-04-24T23:49:00.864867+0000","last_clean":"2024-04-24T23:49:00.864867+0000","last_became_active":"2024-04-24T23:48:55.496715+0000","last_became_peered":"2024-04-24T23:48:55.496715+0000","last_unstale":"2024-04-24T23:49:00.864867+0000","last_undegraded":"2024-04-24T23:49:00.864867+0000","last_fullsized":"2024-04-24T23:49:00.864867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526777+0000","last_change":"2024-04-24T23:48:55.505719+0000","last_active":"2024-04-24T23:49:00.526777+0000","last_peered":"2024-04-24T23:49:00.526777+0000","last_clean":"2024-04-24T23:49:00.526777+0000","last_became_active":"2024-04-24T23:48:55.505418+0000","last_became_peered":"2024-04-24T23:48:55.505418+0000","last_unstale":"2024-04-24T23:49:00.526777+0000","last_undegraded":"2024-04-24T23:49:00.526777+0000","last_fullsized":"2024-04-24T23:49:00.526777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509327+0000","last_change":"2024-04-24T23:48:55.492403+0000","last_active":"2024-04-24T23:49:00.509327+0000","last_peered":"2024-04-24T23:49:00.509327+0000","last_clean":"2024-04-24T23:49:00.509327+0000","last_became_active":"2024-04-24T23:48:55.492112+0000","last_became_peered":"2024-04-24T23:48:55.492112+0000","last_unstale":"2024-04-24T23:49:00.509327+0000","last_undegraded":"2024-04-24T23:49:00.509327+0000","last_fullsized":"2024-04-24T23:49:00.509327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864731+0000","last_change":"2024-04-24T23:48:55.488480+0000","last_active":"2024-04-24T23:49:00.864731+0000","last_peered":"2024-04-24T23:49:00.864731+0000","last_clean":"2024-04-24T23:49:00.864731+0000","last_became_active":"2024-04-24T23:48:55.488134+0000","last_became_peered":"2024-04-24T23:48:55.488134+0000","last_unstale":"2024-04-24T23:49:00.864731+0000","last_undegraded":"2024-04-24T23:49:00.864731+0000","last_fullsized":"2024-04-24T23:49:00.864731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:55.758743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526760+0000","last_change":"2024-04-24T23:48:55.507589+0000","last_active":"2024-04-24T23:49:00.526760+0000","last_peered":"2024-04-24T23:49:00.526760+0000","last_clean":"2024-04-24T23:49:00.526760+0000","last_became_active":"2024-04-24T23:48:55.507281+0000","last_became_peered":"2024-04-24T23:48:55.507281+0000","last_unstale":"2024-04-24T23:49:00.526760+0000","last_undegraded":"2024-04-24T23:49:00.526760+0000","last_fullsized":"2024-04-24T23:49:00.526760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:51.930595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863920+0000","last_change":"2024-04-24T23:48:55.498917+0000","last_active":"2024-04-24T23:49:00.863920+0000","last_peered":"2024-04-24T23:49:00.863920+0000","last_clean":"2024-04-24T23:49:00.863920+0000","last_became_active":"2024-04-24T23:48:55.498627+0000","last_became_peered":"2024-04-24T23:48:55.498627+0000","last_unstale":"2024-04-24T23:49:00.863920+0000","last_undegraded":"2024-04-24T23:49:00.863920+0000","last_fullsized":"2024-04-24T23:49:00.863920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:55.758743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526124+0000","last_change":"2024-04-24T23:48:55.507449+0000","last_active":"2024-04-24T23:49:00.526124+0000","last_peered":"2024-04-24T23:49:00.526124+0000","last_clean":"2024-04-24T23:49:00.526124+0000","last_became_active":"2024-04-24T23:48:55.507151+0000","last_became_peered":"2024-04-24T23:48:55.507151+0000","last_unstale":"2024-04-24T23:49:00.526124+0000","last_undegraded":"2024-04-24T23:49:00.526124+0000","last_fullsized":"2024-04-24T23:49:00.526124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:46.889872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864334+0000","last_change":"2024-04-24T23:48:55.511263+0000","last_active":"2024-04-24T23:49:00.864334+0000","last_peered":"2024-04-24T23:49:00.864334+0000","last_clean":"2024-04-24T23:49:00.864334+0000","last_became_active":"2024-04-24T23:48:55.510422+0000","last_became_peered":"2024-04-24T23:48:55.510422+0000","last_unstale":"2024-04-24T23:49:00.864334+0000","last_undegraded":"2024-04-24T23:49:00.864334+0000","last_fullsized":"2024-04-24T23:49:00.864334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:47.627183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864769+0000","last_change":"2024-04-24T23:48:55.508355+0000","last_active":"2024-04-24T23:49:00.864769+0000","last_peered":"2024-04-24T23:49:00.864769+0000","last_clean":"2024-04-24T23:49:00.864769+0000","last_became_active":"2024-04-24T23:48:55.506836+0000","last_became_peered":"2024-04-24T23:48:55.506836+0000","last_unstale":"2024-04-24T23:49:00.864769+0000","last_undegraded":"2024-04-24T23:49:00.864769+0000","last_fullsized":"2024-04-24T23:49:00.864769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:53:38.263114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864802+0000","last_change":"2024-04-24T23:48:55.507774+0000","last_active":"2024-04-24T23:49:00.864802+0000","last_peered":"2024-04-24T23:49:00.864802+0000","last_clean":"2024-04-24T23:49:00.864802+0000","last_became_active":"2024-04-24T23:48:55.506816+0000","last_became_peered":"2024-04-24T23:48:55.506816+0000","last_unstale":"2024-04-24T23:49:00.864802+0000","last_undegraded":"2024-04-24T23:49:00.864802+0000","last_fullsized":"2024-04-24T23:49:00.864802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510851+0000","last_change":"2024-04-24T23:48:55.492334+0000","last_active":"2024-04-24T23:49:00.510851+0000","last_peered":"2024-04-24T23:49:00.510851+0000","last_clean":"2024-04-24T23:49:00.510851+0000","last_became_active":"2024-04-24T23:48:55.492065+0000","last_became_peered":"2024-04-24T23:48:55.492065+0000","last_unstale":"2024-04-24T23:49:00.510851+0000","last_undegraded":"2024-04-24T23:49:00.510851+0000","last_fullsized":"2024-04-24T23:49:00.510851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:42.483054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863835+0000","last_change":"2024-04-24T23:48:55.495084+0000","last_active":"2024-04-24T23:49:00.863835+0000","last_peered":"2024-04-24T23:49:00.863835+0000","last_clean":"2024-04-24T23:49:00.863835+0000","last_became_active":"2024-04-24T23:48:55.494776+0000","last_became_peered":"2024-04-24T23:48:55.494776+0000","last_unstale":"2024-04-24T23:49:00.863835+0000","last_undegraded":"2024-04-24T23:49:00.863835+0000","last_fullsized":"2024-04-24T23:49:00.863835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:32.416739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865234+0000","last_change":"2024-04-24T23:48:55.511141+0000","last_active":"2024-04-24T23:49:00.865234+0000","last_peered":"2024-04-24T23:49:00.865234+0000","last_clean":"2024-04-24T23:49:00.865234+0000","last_became_active":"2024-04-24T23:48:55.510836+0000","last_became_peered":"2024-04-24T23:48:55.510836+0000","last_unstale":"2024-04-24T23:49:00.865234+0000","last_undegraded":"2024-04-24T23:49:00.865234+0000","last_fullsized":"2024-04-24T23:49:00.865234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:18.684092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863787+0000","last_change":"2024-04-24T23:48:55.508632+0000","last_active":"2024-04-24T23:49:00.863787+0000","last_peered":"2024-04-24T23:49:00.863787+0000","last_clean":"2024-04-24T23:49:00.863787+0000","last_became_active":"2024-04-24T23:48:55.507448+0000","last_became_peered":"2024-04-24T23:48:55.507448+0000","last_unstale":"2024-04-24T23:49:00.863787+0000","last_undegraded":"2024-04-24T23:49:00.863787+0000","last_fullsized":"2024-04-24T23:49:00.863787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:46.889872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508815+0000","last_change":"2024-04-24T23:48:55.508181+0000","last_active":"2024-04-24T23:49:00.508815+0000","last_peered":"2024-04-24T23:49:00.508815+0000","last_clean":"2024-04-24T23:49:00.508815+0000","last_became_active":"2024-04-24T23:48:55.507877+0000","last_became_peered":"2024-04-24T23:48:55.507877+0000","last_unstale":"2024-04-24T23:49:00.508815+0000","last_undegraded":"2024-04-24T23:49:00.508815+0000","last_fullsized":"2024-04-24T23:49:00.508815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864457+0000","last_change":"2024-04-24T23:48:55.497468+0000","last_active":"2024-04-24T23:49:00.864457+0000","last_peered":"2024-04-24T23:49:00.864457+0000","last_clean":"2024-04-24T23:49:00.864457+0000","last_became_active":"2024-04-24T23:48:55.496754+0000","last_became_peered":"2024-04-24T23:48:55.496754+0000","last_unstale":"2024-04-24T23:49:00.864457+0000","last_undegraded":"2024-04-24T23:49:00.864457+0000","last_fullsized":"2024-04-24T23:49:00.864457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:58.209422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526144+0000","last_change":"2024-04-24T23:48:55.492306+0000","last_active":"2024-04-24T23:49:00.526144+0000","last_peered":"2024-04-24T23:49:00.526144+0000","last_clean":"2024-04-24T23:49:00.526144+0000","last_became_active":"2024-04-24T23:48:55.492008+0000","last_became_peered":"2024-04-24T23:48:55.492008+0000","last_unstale":"2024-04-24T23:49:00.526144+0000","last_undegraded":"2024-04-24T23:49:00.526144+0000","last_fullsized":"2024-04-24T23:49:00.526144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:55.758743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510496+0000","last_change":"2024-04-24T23:48:55.501767+0000","last_active":"2024-04-24T23:49:00.510496+0000","last_peered":"2024-04-24T23:49:00.510496+0000","last_clean":"2024-04-24T23:49:00.510496+0000","last_became_active":"2024-04-24T23:48:55.500776+0000","last_became_peered":"2024-04-24T23:48:55.500776+0000","last_unstale":"2024-04-24T23:49:00.510496+0000","last_undegraded":"2024-04-24T23:49:00.510496+0000","last_fullsized":"2024-04-24T23:49:00.510496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:04.085761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864571+0000","last_change":"2024-04-24T23:48:55.511220+0000","last_active":"2024-04-24T23:49:00.864571+0000","last_peered":"2024-04-24T23:49:00.864571+0000","last_clean":"2024-04-24T23:49:00.864571+0000","last_became_active":"2024-04-24T23:48:55.510937+0000","last_became_peered":"2024-04-24T23:48:55.510937+0000","last_unstale":"2024-04-24T23:49:00.864571+0000","last_undegraded":"2024-04-24T23:49:00.864571+0000","last_fullsized":"2024-04-24T23:49:00.864571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864622+0000","last_change":"2024-04-24T23:48:55.503843+0000","last_active":"2024-04-24T23:49:00.864622+0000","last_peered":"2024-04-24T23:49:00.864622+0000","last_clean":"2024-04-24T23:49:00.864622+0000","last_became_active":"2024-04-24T23:48:55.503569+0000","last_became_peered":"2024-04-24T23:48:55.503569+0000","last_unstale":"2024-04-24T23:49:00.864622+0000","last_undegraded":"2024-04-24T23:49:00.864622+0000","last_fullsized":"2024-04-24T23:49:00.864622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:46.889872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511357+0000","last_change":"2024-04-24T23:48:55.509090+0000","last_active":"2024-04-24T23:49:00.511357+0000","last_peered":"2024-04-24T23:49:00.511357+0000","last_clean":"2024-04-24T23:49:00.511357+0000","last_became_active":"2024-04-24T23:48:55.508730+0000","last_became_peered":"2024-04-24T23:48:55.508730+0000","last_unstale":"2024-04-24T23:49:00.511357+0000","last_undegraded":"2024-04-24T23:49:00.511357+0000","last_fullsized":"2024-04-24T23:49:00.511357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:18.684092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.525876+0000","last_change":"2024-04-24T23:48:55.498351+0000","last_active":"2024-04-24T23:49:00.525876+0000","last_peered":"2024-04-24T23:49:00.525876+0000","last_clean":"2024-04-24T23:49:00.525876+0000","last_became_active":"2024-04-24T23:48:55.497694+0000","last_became_peered":"2024-04-24T23:48:55.497694+0000","last_unstale":"2024-04-24T23:49:00.525876+0000","last_undegraded":"2024-04-24T23:49:00.525876+0000","last_fullsized":"2024-04-24T23:49:00.525876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509595+0000","last_change":"2024-04-24T23:48:55.501677+0000","last_active":"2024-04-24T23:49:00.509595+0000","last_peered":"2024-04-24T23:49:00.509595+0000","last_clean":"2024-04-24T23:49:00.509595+0000","last_became_active":"2024-04-24T23:48:55.501359+0000","last_became_peered":"2024-04-24T23:48:55.501359+0000","last_unstale":"2024-04-24T23:49:00.509595+0000","last_undegraded":"2024-04-24T23:49:00.509595+0000","last_fullsized":"2024-04-24T23:49:00.509595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:47.627183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864613+0000","last_change":"2024-04-24T23:48:55.511089+0000","last_active":"2024-04-24T23:49:00.864613+0000","last_peered":"2024-04-24T23:49:00.864613+0000","last_clean":"2024-04-24T23:49:00.864613+0000","last_became_active":"2024-04-24T23:48:55.510818+0000","last_became_peered":"2024-04-24T23:48:55.510818+0000","last_unstale":"2024-04-24T23:49:00.864613+0000","last_undegraded":"2024-04-24T23:49:00.864613+0000","last_fullsized":"2024-04-24T23:49:00.864613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:03:28.472120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509524+0000","last_change":"2024-04-24T23:48:55.492973+0000","last_active":"2024-04-24T23:49:00.509524+0000","last_peered":"2024-04-24T23:49:00.509524+0000","last_clean":"2024-04-24T23:49:00.509524+0000","last_became_active":"2024-04-24T23:48:55.492713+0000","last_became_peered":"2024-04-24T23:48:55.492713+0000","last_unstale":"2024-04-24T23:49:00.509524+0000","last_undegraded":"2024-04-24T23:49:00.509524+0000","last_fullsized":"2024-04-24T23:49:00.509524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:10.290840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864561+0000","last_change":"2024-04-24T23:48:55.508975+0000","last_active":"2024-04-24T23:49:00.864561+0000","last_peered":"2024-04-24T23:49:00.864561+0000","last_clean":"2024-04-24T23:49:00.864561+0000","last_became_active":"2024-04-24T23:48:55.507970+0000","last_became_peered":"2024-04-24T23:48:55.507970+0000","last_unstale":"2024-04-24T23:49:00.864561+0000","last_undegraded":"2024-04-24T23:49:00.864561+0000","last_fullsized":"2024-04-24T23:49:00.864561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:10.235664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864614+0000","last_change":"2024-04-24T23:48:55.500274+0000","last_active":"2024-04-24T23:49:00.864614+0000","last_peered":"2024-04-24T23:49:00.864614+0000","last_clean":"2024-04-24T23:49:00.864614+0000","last_became_active":"2024-04-24T23:48:55.500004+0000","last_became_peered":"2024-04-24T23:48:55.500004+0000","last_unstale":"2024-04-24T23:49:00.864614+0000","last_undegraded":"2024-04-24T23:49:00.864614+0000","last_fullsized":"2024-04-24T23:49:00.864614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:02:47.989664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509710+0000","last_change":"2024-04-24T23:48:55.499555+0000","last_active":"2024-04-24T23:49:00.509710+0000","last_peered":"2024-04-24T23:49:00.509710+0000","last_clean":"2024-04-24T23:49:00.509710+0000","last_became_active":"2024-04-24T23:48:55.499163+0000","last_became_peered":"2024-04-24T23:48:55.499163+0000","last_unstale":"2024-04-24T23:49:00.509710+0000","last_undegraded":"2024-04-24T23:49:00.509710+0000","last_fullsized":"2024-04-24T23:49:00.509710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511249+0000","last_change":"2024-04-24T23:48:55.493200+0000","last_active":"2024-04-24T23:49:00.511249+0000","last_peered":"2024-04-24T23:49:00.511249+0000","last_clean":"2024-04-24T23:49:00.511249+0000","last_became_active":"2024-04-24T23:48:55.491655+0000","last_became_peered":"2024-04-24T23:48:55.491655+0000","last_unstale":"2024-04-24T23:49:00.511249+0000","last_undegraded":"2024-04-24T23:49:00.511249+0000","last_fullsized":"2024-04-24T23:49:00.511249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:04.801548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:49:19.911 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-24T23:49:19.911 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-24T23:49:19.911 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-04-24T23:49:19.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-04-24T23:49:19.912 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-04-24T23:49:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.047+0000 7f46832bd640 1 Processor -- start 2024-04-24T23:49:20.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.048+0000 7f46832bd640 1 -- start start 2024-04-24T23:49:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.048+0000 7f46832bd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f467c1457b0 0x7f467c0ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.048+0000 7f46832bd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f467c15f990 0x7f467c0cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.048+0000 7f46832bd640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f467c15fe90 con 0x7f467c144b20 2024-04-24T23:49:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.048+0000 7f46832bd640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f467c06ea50 con 0x7f467c15f990 2024-04-24T23:49:20.048 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.048+0000 7f46832bd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f467c160170 con 0x7f467c1457b0 2024-04-24T23:49:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.049+0000 7f4680858640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f467c1457b0 0x7f467c0ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.049+0000 7f468185a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f467c15f990 0x7f467c0cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.049+0000 7f4680858640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f467c1457b0 0x7f467c0ce680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:41330/0 (socket says 172.21.15.16:41330) 2024-04-24T23:49:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.049+0000 7f4681059640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f467c144b20 0x7f467c144f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:41224/0 (socket says 172.21.15.16:41224) 2024-04-24T23:49:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.049+0000 7f4680858640 1 -- 172.21.15.16:0/1051156084 learned_addr learned my addr 172.21.15.16:0/1051156084 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.049+0000 7f468185a640 1 --2- 172.21.15.16:0/1051156084 >> v2:172.21.15.156:3300/0 conn(0x7f467c15f990 0x7f467c0cebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:34692/0 (socket says 172.21.15.16:34692) 2024-04-24T23:49:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.049+0000 7f4680858640 1 -- 172.21.15.16:0/1051156084 >> v2:172.21.15.156:3300/0 conn(0x7f467c15f990 msgr2=0x7f467c0cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.049+0000 7f4680858640 1 --2- 172.21.15.16:0/1051156084 >> v2:172.21.15.156:3300/0 conn(0x7f467c15f990 0x7f467c0cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.049+0000 7f4680858640 1 -- 172.21.15.16:0/1051156084 >> v1:172.21.15.156:6789/0 conn(0x7f467c144b20 legacy=0x7f467c144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.049+0000 7f4680858640 1 -- 172.21.15.16:0/1051156084 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467c160450 con 0x7f467c1457b0 2024-04-24T23:49:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.050+0000 7f4680858640 1 --2- 172.21.15.16:0/1051156084 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f467c1457b0 0x7f467c0ce680 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4670009870 tx=0x7f467002a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d524f6c086f74044 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.050+0000 7f467bfff640 1 -- 172.21.15.16:0/1051156084 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4670037070 con 0x7f467c1457b0 2024-04-24T23:49:20.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.050+0000 7f467bfff640 1 -- 172.21.15.16:0/1051156084 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4670032440 con 0x7f467c1457b0 2024-04-24T23:49:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.050+0000 7f467bfff640 1 -- 172.21.15.16:0/1051156084 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4670032810 con 0x7f467c1457b0 2024-04-24T23:49:20.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.051+0000 7f46832bd640 1 -- 172.21.15.16:0/1051156084 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f467c1457b0 msgr2=0x7f467c0ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.051+0000 7f46832bd640 1 --2- 172.21.15.16:0/1051156084 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f467c1457b0 0x7f467c0ce680 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4670009870 tx=0x7f467002a1c0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.051+0000 7f46832bd640 1 -- 172.21.15.16:0/1051156084 shutdown_connections 2024-04-24T23:49:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.051+0000 7f46832bd640 1 --2- 172.21.15.16:0/1051156084 >> v2:172.21.15.156:3300/0 conn(0x7f467c15f990 0x7f467c0cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.051+0000 7f46832bd640 1 --2- 172.21.15.16:0/1051156084 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f467c1457b0 0x7f467c0ce680 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.051+0000 7f46832bd640 1 -- 172.21.15.16:0/1051156084 >> 172.21.15.16:0/1051156084 conn(0x7f467c0b2000 msgr2=0x7f467c0cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.051+0000 7f46832bd640 1 -- 172.21.15.16:0/1051156084 shutdown_connections 2024-04-24T23:49:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.051+0000 7f46832bd640 1 -- 172.21.15.16:0/1051156084 wait complete. 2024-04-24T23:49:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.052+0000 7f46832bd640 1 Processor -- start 2024-04-24T23:49:20.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.052+0000 7f46832bd640 1 -- start start 2024-04-24T23:49:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f46832bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f467c144b20 0x7f467c138630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f46832bd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f467c1457b0 0x7f467c1406e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f46832bd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f467c15f990 0x7f467c1591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f46832bd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f467c0b3390 con 0x7f467c144b20 2024-04-24T23:49:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f46832bd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f467c06d4b0 con 0x7f467c15f990 2024-04-24T23:49:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f46832bd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f467c06da40 con 0x7f467c1457b0 2024-04-24T23:49:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f4681059640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f467c144b20 0x7f467c138630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f4681059640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f467c144b20 0x7f467c138630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48272/0 (socket says 172.21.15.16:48272) 2024-04-24T23:49:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f4681059640 1 -- 172.21.15.16:0/2087431828 learned_addr learned my addr 172.21.15.16:0/2087431828 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f4680858640 1 --2- 172.21.15.16:0/2087431828 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f467c1457b0 0x7f467c1406e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f468185a640 1 --2- 172.21.15.16:0/2087431828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f467c15f990 0x7f467c1591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f4681059640 1 -- 172.21.15.16:0/2087431828 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f467c1457b0 msgr2=0x7f467c1406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f4681059640 1 --2- 172.21.15.16:0/2087431828 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f467c1457b0 0x7f467c1406e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f4681059640 1 -- 172.21.15.16:0/2087431828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f467c15f990 msgr2=0x7f467c1591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f4681059640 1 --2- 172.21.15.16:0/2087431828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f467c15f990 0x7f467c1591c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f4681059640 1 -- 172.21.15.16:0/2087431828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f467c157e80 con 0x7f467c144b20 2024-04-24T23:49:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f4681059640 1 --2- 172.21.15.16:0/2087431828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f467c144b20 0x7f467c138630 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f467400c400 tx=0x7f467400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f4679ffb640 1 -- 172.21.15.16:0/2087431828 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46740074e0 con 0x7f467c144b20 2024-04-24T23:49:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f4679ffb640 1 -- 172.21.15.16:0/2087431828 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4674007680 con 0x7f467c144b20 2024-04-24T23:49:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.053+0000 7f46832bd640 1 -- 172.21.15.16:0/2087431828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f467c1f1fb0 con 0x7f467c144b20 2024-04-24T23:49:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.054+0000 7f4679ffb640 1 -- 172.21.15.16:0/2087431828 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4674004a00 con 0x7f467c144b20 2024-04-24T23:49:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.054+0000 7f46832bd640 1 -- 172.21.15.16:0/2087431828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f467c1f2570 con 0x7f467c144b20 2024-04-24T23:49:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.055+0000 7f46832bd640 1 -- 172.21.15.16:0/2087431828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f467c0731f0 con 0x7f467c144b20 2024-04-24T23:49:20.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.055+0000 7f4679ffb640 1 -- 172.21.15.16:0/2087431828 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f467401a020 con 0x7f467c144b20 2024-04-24T23:49:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.056+0000 7f4679ffb640 1 --2- 172.21.15.16:0/2087431828 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f466806be30 0x7f466806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.056+0000 7f4679ffb640 1 -- 172.21.15.16:0/2087431828 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f467408a7d0 con 0x7f467c144b20 2024-04-24T23:49:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.056+0000 7f4680858640 1 --2- 172.21.15.16:0/2087431828 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f466806be30 0x7f466806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.056+0000 7f4680858640 1 --2- 172.21.15.16:0/2087431828 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f466806be30 0x7f466806e2f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4670034000 tx=0x7f467002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:20.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.060+0000 7f4679ffb640 1 -- 172.21.15.16:0/2087431828 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f467c157e80 con 0x7f467c144b20 2024-04-24T23:49:20.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.293+0000 7f46832bd640 1 -- 172.21.15.16:0/2087431828 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f467c04f5e0 con 0x7f467c144b20 2024-04-24T23:49:20.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.296+0000 7f4679ffb640 1 -- 172.21.15.16:0/2087431828 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+172946 (secure 0 0 0) 0x7f4674058880 con 0x7f467c144b20 2024-04-24T23:49:20.298 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:20.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.298+0000 7f46832bd640 1 -- 172.21.15.16:0/2087431828 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f466806be30 msgr2=0x7f466806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.298+0000 7f46832bd640 1 --2- 172.21.15.16:0/2087431828 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f466806be30 0x7f466806e2f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4670034000 tx=0x7f467002e000 comp rx=0 tx=0).stop 2024-04-24T23:49:20.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.299+0000 7f46832bd640 1 -- 172.21.15.16:0/2087431828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f467c144b20 msgr2=0x7f467c138630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.299+0000 7f46832bd640 1 --2- 172.21.15.16:0/2087431828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f467c144b20 0x7f467c138630 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f467400c400 tx=0x7f467400c8c0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.299+0000 7f46832bd640 1 -- 172.21.15.16:0/2087431828 shutdown_connections 2024-04-24T23:49:20.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.299+0000 7f46832bd640 1 --2- 172.21.15.16:0/2087431828 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f466806be30 0x7f466806e2f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.299+0000 7f46832bd640 1 --2- 172.21.15.16:0/2087431828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f467c15f990 0x7f467c1591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.299+0000 7f46832bd640 1 --2- 172.21.15.16:0/2087431828 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f467c1457b0 0x7f467c1406e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.299+0000 7f46832bd640 1 --2- 172.21.15.16:0/2087431828 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f467c144b20 0x7f467c138630 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.299+0000 7f46832bd640 1 -- 172.21.15.16:0/2087431828 >> 172.21.15.16:0/2087431828 conn(0x7f467c0b2000 msgr2=0x7f467c0b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.299+0000 7f46832bd640 1 -- 172.21.15.16:0/2087431828 shutdown_connections 2024-04-24T23:49:20.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.299+0000 7f46832bd640 1 -- 172.21.15.16:0/2087431828 wait complete. 2024-04-24T23:49:20.309 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":5,"flags":0,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":3478836820},{"type":"v1","addr":"172.21.15.156:6833","nonce":3478836820}]},"active_addr":"172.21.15.156:6833/3478836820","active_change":"2024-04-24T23:48:35.108687+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4106,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":1858871075}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":2948218405}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":1433980675}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":2739097070}]}]} 2024-04-24T23:49:20.312 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-04-24T23:49:20.312 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-04-24T23:49:20.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.413+0000 7fd9b0190640 1 Processor -- start 2024-04-24T23:49:20.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.414+0000 7fd9b0190640 1 -- start start 2024-04-24T23:49:20.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.414+0000 7fd9b0190640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd9a815e020 0x7fd9a80cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.414+0000 7fd9b0190640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9a80b3390 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.414+0000 7fd9b0190640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9a806bfc0 con 0x7fd9a80d1fa0 2024-04-24T23:49:20.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.414+0000 7fd9b0190640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd9a806c2a0 con 0x7fd9a815e020 2024-04-24T23:49:20.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.414+0000 7fd9adf2c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd9a815e020 0x7fd9a80cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.414+0000 7fd9ad72b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd9a80b2bb0 0x7fd9a80cf7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50604/0 (socket says 172.21.15.16:50604) 2024-04-24T23:49:20.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.414+0000 7fd9ad72b640 1 -- 172.21.15.16:0/597991284 learned_addr learned my addr 172.21.15.16:0/597991284 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:20.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.415+0000 7fd9adf2c640 1 -- 172.21.15.16:0/597991284 >> v1:172.21.15.156:6789/0 conn(0x7fd9a80d1fa0 legacy=0x7fd9a80d23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.415+0000 7fd9adf2c640 1 -- 172.21.15.16:0/597991284 >> v1:172.21.15.16:6789/0 conn(0x7fd9a80b2bb0 legacy=0x7fd9a80cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.415+0000 7fd9adf2c640 1 -- 172.21.15.16:0/597991284 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a806c580 con 0x7fd9a815e020 2024-04-24T23:49:20.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.415+0000 7fd9acf2a640 1 -- 172.21.15.16:0/597991284 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3500030028 0 0) 0x7fd9a80b3390 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.416+0000 7fd9adf2c640 1 --2- 172.21.15.16:0/597991284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a815e020 0x7fd9a80cf270 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd998009d90 tx=0x7fd99802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5696534b741515bd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:20.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.416+0000 7fd9acf2a640 1 -- 172.21.15.16:0/597991284 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd998007740 con 0x7fd9a815e020 2024-04-24T23:49:20.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.416+0000 7fd9acf2a640 1 -- 172.21.15.16:0/597991284 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9980078e0 con 0x7fd9a815e020 2024-04-24T23:49:20.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.416+0000 7fd9acf2a640 1 -- 172.21.15.16:0/597991284 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd998007cb0 con 0x7fd9a815e020 2024-04-24T23:49:20.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.416+0000 7fd9b0190640 1 -- 172.21.15.16:0/597991284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a815e020 msgr2=0x7fd9a80cf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.416+0000 7fd9b0190640 1 --2- 172.21.15.16:0/597991284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a815e020 0x7fd9a80cf270 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd998009d90 tx=0x7fd99802a660 comp rx=0 tx=0).stop 2024-04-24T23:49:20.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.416+0000 7fd9b0190640 1 -- 172.21.15.16:0/597991284 shutdown_connections 2024-04-24T23:49:20.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.416+0000 7fd9b0190640 1 --2- 172.21.15.16:0/597991284 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a815e020 0x7fd9a80cf270 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.416+0000 7fd9b0190640 1 -- 172.21.15.16:0/597991284 >> 172.21.15.16:0/597991284 conn(0x7fd9a80d3e10 msgr2=0x7fd9a80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:20.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.416+0000 7fd9b0190640 1 -- 172.21.15.16:0/597991284 shutdown_connections 2024-04-24T23:49:20.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.416+0000 7fd9b0190640 1 -- 172.21.15.16:0/597991284 wait complete. 2024-04-24T23:49:20.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.417+0000 7fd9b0190640 1 Processor -- start 2024-04-24T23:49:20.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.418+0000 7fd9b0190640 1 -- start start 2024-04-24T23:49:20.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.418+0000 7fd9b0190640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a80b2bb0 0x7fd9a815c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.418+0000 7fd9b0190640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9a80d1fa0 0x7fd9a81e82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.418+0000 7fd9b0190640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd9a815e020 0x7fd9a80d2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.418+0000 7fd9b0190640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a815e400 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.418+0000 7fd9b0190640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd9a80d1a80 con 0x7fd9a815e020 2024-04-24T23:49:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.418+0000 7fd9b0190640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fd9a80cfec0 con 0x7fd9a80d1fa0 2024-04-24T23:49:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.418+0000 7fd9adf2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a80b2bb0 0x7fd9a815c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.418+0000 7fd9adf2c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a80b2bb0 0x7fd9a815c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48304/0 (socket says 172.21.15.16:48304) 2024-04-24T23:49:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.418+0000 7fd9adf2c640 1 -- 172.21.15.16:0/3280075905 learned_addr learned my addr 172.21.15.16:0/3280075905 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.418+0000 7fd9ae72d640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd9a815e020 0x7fd9a80d2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.419+0000 7fd9ad72b640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9a80d1fa0 0x7fd9a81e82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.419+0000 7fd9adf2c640 1 -- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9a80d1fa0 msgr2=0x7fd9a81e82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.419+0000 7fd9adf2c640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9a80d1fa0 0x7fd9a81e82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.419+0000 7fd9adf2c640 1 -- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd9a815e020 msgr2=0x7fd9a80d2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.419+0000 7fd9adf2c640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd9a815e020 0x7fd9a80d2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.419+0000 7fd9adf2c640 1 -- 172.21.15.16:0/3280075905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9a80ac120 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.419+0000 7fd9ae72d640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd9a815e020 0x7fd9a80d2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.419+0000 7fd9adf2c640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a80b2bb0 0x7fd9a815c750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd998009d90 tx=0x7fd998004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.419+0000 7fd9a67fc640 1 -- 172.21.15.16:0/3280075905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd99803f070 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.419+0000 7fd9a67fc640 1 -- 172.21.15.16:0/3280075905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd998031d00 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.419+0000 7fd9a67fc640 1 -- 172.21.15.16:0/3280075905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd998044660 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.419+0000 7fd9b0190640 1 -- 172.21.15.16:0/3280075905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9a80ac400 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.419+0000 7fd9b0190640 1 -- 172.21.15.16:0/3280075905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9a80acca0 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.420+0000 7fd9b0190640 1 -- 172.21.15.16:0/3280075905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9a80ac5d0 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.421+0000 7fd9a67fc640 1 -- 172.21.15.16:0/3280075905 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd99803b410 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.421+0000 7fd9a67fc640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd99406be30 0x7fd99406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.421+0000 7fd9a67fc640 1 -- 172.21.15.16:0/3280075905 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fd9980acaf0 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.421+0000 7fd9ad72b640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd99406be30 0x7fd99406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.422+0000 7fd9ad72b640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd99406be30 0x7fd99406e2f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd99c007f70 tx=0x7fd99c007f00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:20.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.425+0000 7fd9a67fc640 1 -- 172.21.15.16:0/3280075905 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd99807b0c0 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.612+0000 7fd9b0190640 1 -- 172.21.15.16:0/3280075905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd9a806dfe0 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.613+0000 7fd9a67fc640 1 -- 172.21.15.16:0/3280075905 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18502 (secure 0 0 0) 0x7fd99807a9c0 con 0x7fd9a80b2bb0 2024-04-24T23:49:20.613 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:20.613 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:59.888015+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:20.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.615+0000 7fd9b0190640 1 -- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd99406be30 msgr2=0x7fd99406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.615+0000 7fd9b0190640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd99406be30 0x7fd99406e2f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd99c007f70 tx=0x7fd99c007f00 comp rx=0 tx=0).stop 2024-04-24T23:49:20.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.615+0000 7fd9b0190640 1 -- 172.21.15.16:0/3280075905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a80b2bb0 msgr2=0x7fd9a815c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.615+0000 7fd9b0190640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a80b2bb0 0x7fd9a815c750 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd998009d90 tx=0x7fd998004a40 comp rx=0 tx=0).stop 2024-04-24T23:49:20.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.615+0000 7fd9b0190640 1 -- 172.21.15.16:0/3280075905 shutdown_connections 2024-04-24T23:49:20.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.615+0000 7fd9b0190640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd99406be30 0x7fd99406e2f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.615+0000 7fd9b0190640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd9a815e020 0x7fd9a80d2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.615+0000 7fd9b0190640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd9a80d1fa0 0x7fd9a81e82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.615+0000 7fd9b0190640 1 --2- 172.21.15.16:0/3280075905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd9a80b2bb0 0x7fd9a815c750 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.615+0000 7fd9b0190640 1 -- 172.21.15.16:0/3280075905 >> 172.21.15.16:0/3280075905 conn(0x7fd9a80d3e10 msgr2=0x7fd9a806b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:20.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.615+0000 7fd9b0190640 1 -- 172.21.15.16:0/3280075905 shutdown_connections 2024-04-24T23:49:20.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.615+0000 7fd9b0190640 1 -- 172.21.15.16:0/3280075905 wait complete. 2024-04-24T23:49:20.630 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-04-24T23:49:20.630 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.735+0000 7fdd90aa5640 1 Processor -- start 2024-04-24T23:49:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.735+0000 7fdd90aa5640 1 -- start start 2024-04-24T23:49:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.736+0000 7fdd90aa5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdd8c0d16c0 0x7fdd8c1601f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.736+0000 7fdd90aa5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd8c0705b0 0x7fdd8c06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.736+0000 7fdd90aa5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd8c1607d0 con 0x7fdd8c15bcd0 2024-04-24T23:49:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.736+0000 7fdd90aa5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fdd8c06f660 con 0x7fdd8c0d16c0 2024-04-24T23:49:20.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.736+0000 7fdd90aa5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fdd8c06ed30 con 0x7fdd8c0705b0 2024-04-24T23:49:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.736+0000 7fdd89d9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdd8c0d16c0 0x7fdd8c1601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.736+0000 7fdd8ad9d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd8c0705b0 0x7fdd8c06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.737+0000 7fdd89d9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdd8c0d16c0 0x7fdd8c1601f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:34720/0 (socket says 172.21.15.16:34720) 2024-04-24T23:49:20.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.737+0000 7fdd89d9b640 1 -- 172.21.15.16:0/659839445 learned_addr learned my addr 172.21.15.16:0/659839445 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.737+0000 7fdd8a59c640 1 --1- 172.21.15.16:0/659839445 >> v1:172.21.15.156:6789/0 conn(0x7fdd8c15bcd0 0x7fdd8c0d0fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:41252/0 (socket says 172.21.15.16:41252) 2024-04-24T23:49:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.737+0000 7fdd8ad9d640 1 -- 172.21.15.16:0/659839445 >> v2:172.21.15.156:3300/0 conn(0x7fdd8c0d16c0 msgr2=0x7fdd8c1601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.737+0000 7fdd8ad9d640 1 --2- 172.21.15.16:0/659839445 >> v2:172.21.15.156:3300/0 conn(0x7fdd8c0d16c0 0x7fdd8c1601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.737+0000 7fdd8ad9d640 1 -- 172.21.15.16:0/659839445 >> v1:172.21.15.156:6789/0 conn(0x7fdd8c15bcd0 legacy=0x7fdd8c0d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.737+0000 7fdd8ad9d640 1 -- 172.21.15.16:0/659839445 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8c06f010 con 0x7fdd8c0705b0 2024-04-24T23:49:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.737+0000 7fdd89d9b640 1 --2- 172.21.15.16:0/659839445 >> v2:172.21.15.156:3300/0 conn(0x7fdd8c0d16c0 0x7fdd8c1601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:20.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.738+0000 7fdd8ad9d640 1 --2- 172.21.15.16:0/659839445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd8c0705b0 0x7fdd8c06faf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdd840097a0 tx=0x7fdd8402a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5eb98126ceb7b209 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:20.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.738+0000 7fdd8959a640 1 -- 172.21.15.16:0/659839445 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd84037070 con 0x7fdd8c0705b0 2024-04-24T23:49:20.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.738+0000 7fdd8959a640 1 -- 172.21.15.16:0/659839445 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd8402adb0 con 0x7fdd8c0705b0 2024-04-24T23:49:20.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.738+0000 7fdd8959a640 1 -- 172.21.15.16:0/659839445 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd84032660 con 0x7fdd8c0705b0 2024-04-24T23:49:20.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.738+0000 7fdd90aa5640 1 -- 172.21.15.16:0/659839445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd8c0705b0 msgr2=0x7fdd8c06faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.738+0000 7fdd90aa5640 1 --2- 172.21.15.16:0/659839445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd8c0705b0 0x7fdd8c06faf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdd840097a0 tx=0x7fdd8402a060 comp rx=0 tx=0).stop 2024-04-24T23:49:20.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.738+0000 7fdd90aa5640 1 -- 172.21.15.16:0/659839445 shutdown_connections 2024-04-24T23:49:20.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.738+0000 7fdd90aa5640 1 --2- 172.21.15.16:0/659839445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd8c0705b0 0x7fdd8c06faf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.738+0000 7fdd90aa5640 1 --2- 172.21.15.16:0/659839445 >> v2:172.21.15.156:3300/0 conn(0x7fdd8c0d16c0 0x7fdd8c1601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.738+0000 7fdd90aa5640 1 -- 172.21.15.16:0/659839445 >> 172.21.15.16:0/659839445 conn(0x7fdd8c06b4d0 msgr2=0x7fdd8c0b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:20.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.739+0000 7fdd90aa5640 1 -- 172.21.15.16:0/659839445 shutdown_connections 2024-04-24T23:49:20.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.739+0000 7fdd90aa5640 1 -- 172.21.15.16:0/659839445 wait complete. 2024-04-24T23:49:20.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.740+0000 7fdd90aa5640 1 Processor -- start 2024-04-24T23:49:20.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.740+0000 7fdd90aa5640 1 -- start start 2024-04-24T23:49:20.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.740+0000 7fdd90aa5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd8c0705b0 0x7fdd8c0cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.740+0000 7fdd90aa5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c0d16c0 0x7fdd8c157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd90aa5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd8c15bcd0 0x7fdd8c0b5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd90aa5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8c0d09d0 con 0x7fdd8c15bcd0 2024-04-24T23:49:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd90aa5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8c15c0b0 con 0x7fdd8c0d16c0 2024-04-24T23:49:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd90aa5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fdd8c0d0b50 con 0x7fdd8c0705b0 2024-04-24T23:49:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd8ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd8c15bcd0 0x7fdd8c0b5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd8ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd8c15bcd0 0x7fdd8c0b5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48306/0 (socket says 172.21.15.16:48306) 2024-04-24T23:49:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd8ad9d640 1 -- 172.21.15.16:0/2511269406 learned_addr learned my addr 172.21.15.16:0/2511269406 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd8a59c640 1 --2- 172.21.15.16:0/2511269406 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd8c0705b0 0x7fdd8c0cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd89d9b640 1 --2- 172.21.15.16:0/2511269406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c0d16c0 0x7fdd8c157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd8ad9d640 1 -- 172.21.15.16:0/2511269406 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd8c0705b0 msgr2=0x7fdd8c0cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd8ad9d640 1 --2- 172.21.15.16:0/2511269406 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd8c0705b0 0x7fdd8c0cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd8ad9d640 1 -- 172.21.15.16:0/2511269406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c0d16c0 msgr2=0x7fdd8c157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd8ad9d640 1 --2- 172.21.15.16:0/2511269406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c0d16c0 0x7fdd8c157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd8ad9d640 1 -- 172.21.15.16:0/2511269406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd8c1f1f50 con 0x7fdd8c15bcd0 2024-04-24T23:49:20.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd8ad9d640 1 --2- 172.21.15.16:0/2511269406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd8c15bcd0 0x7fdd8c0b5cd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdd84035040 tx=0x7fdd840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:20.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd7affd640 1 -- 172.21.15.16:0/2511269406 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd84041070 con 0x7fdd8c15bcd0 2024-04-24T23:49:20.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd7affd640 1 -- 172.21.15.16:0/2511269406 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd8403c460 con 0x7fdd8c15bcd0 2024-04-24T23:49:20.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd7affd640 1 -- 172.21.15.16:0/2511269406 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd84037040 con 0x7fdd8c15bcd0 2024-04-24T23:49:20.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.741+0000 7fdd90aa5640 1 -- 172.21.15.16:0/2511269406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd8c0c8e50 con 0x7fdd8c15bcd0 2024-04-24T23:49:20.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.742+0000 7fdd90aa5640 1 -- 172.21.15.16:0/2511269406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd8c0b5710 con 0x7fdd8c15bcd0 2024-04-24T23:49:20.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.743+0000 7fdd90aa5640 1 -- 172.21.15.16:0/2511269406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd8c06bea0 con 0x7fdd8c15bcd0 2024-04-24T23:49:20.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.743+0000 7fdd7affd640 1 -- 172.21.15.16:0/2511269406 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdd84044070 con 0x7fdd8c15bcd0 2024-04-24T23:49:20.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.743+0000 7fdd7affd640 1 --2- 172.21.15.16:0/2511269406 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdd6c06be30 0x7fdd6c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:20.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.744+0000 7fdd7affd640 1 -- 172.21.15.16:0/2511269406 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdd840ab650 con 0x7fdd8c15bcd0 2024-04-24T23:49:20.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.744+0000 7fdd8a59c640 1 --2- 172.21.15.16:0/2511269406 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdd6c06be30 0x7fdd6c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:20.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.744+0000 7fdd8a59c640 1 --2- 172.21.15.16:0/2511269406 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdd6c06be30 0x7fdd6c06e2f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0b3630 tx=0x7fdd7c001630 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:20.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.748+0000 7fdd7affd640 1 -- 172.21.15.16:0/2511269406 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdd84079700 con 0x7fdd8c15bcd0 2024-04-24T23:49:20.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.936+0000 7fdd90aa5640 1 -- 172.21.15.16:0/2511269406 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd8c04f5e0 con 0x7fdd8c15bcd0 2024-04-24T23:49:20.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.937+0000 7fdd7affd640 1 -- 172.21.15.16:0/2511269406 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18502 (secure 0 0 0) 0x7fdd8407da00 con 0x7fdd8c15bcd0 2024-04-24T23:49:20.937 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:20.937 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:59.888015+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:20.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.939+0000 7fdd90aa5640 1 -- 172.21.15.16:0/2511269406 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdd6c06be30 msgr2=0x7fdd6c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.939+0000 7fdd90aa5640 1 --2- 172.21.15.16:0/2511269406 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdd6c06be30 0x7fdd6c06e2f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0b3630 tx=0x7fdd7c001630 comp rx=0 tx=0).stop 2024-04-24T23:49:20.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.939+0000 7fdd90aa5640 1 -- 172.21.15.16:0/2511269406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd8c15bcd0 msgr2=0x7fdd8c0b5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:20.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.939+0000 7fdd90aa5640 1 --2- 172.21.15.16:0/2511269406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd8c15bcd0 0x7fdd8c0b5cd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fdd84035040 tx=0x7fdd840046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.939+0000 7fdd90aa5640 1 -- 172.21.15.16:0/2511269406 shutdown_connections 2024-04-24T23:49:20.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.939+0000 7fdd90aa5640 1 --2- 172.21.15.16:0/2511269406 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdd6c06be30 0x7fdd6c06e2f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.939+0000 7fdd90aa5640 1 --2- 172.21.15.16:0/2511269406 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd8c15bcd0 0x7fdd8c0b5cd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.939+0000 7fdd90aa5640 1 --2- 172.21.15.16:0/2511269406 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd8c0d16c0 0x7fdd8c157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.939+0000 7fdd90aa5640 1 --2- 172.21.15.16:0/2511269406 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd8c0705b0 0x7fdd8c0cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:20.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.939+0000 7fdd90aa5640 1 -- 172.21.15.16:0/2511269406 >> 172.21.15.16:0/2511269406 conn(0x7fdd8c06b4d0 msgr2=0x7fdd8c15e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:20.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.939+0000 7fdd90aa5640 1 -- 172.21.15.16:0/2511269406 shutdown_connections 2024-04-24T23:49:20.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:20.939+0000 7fdd90aa5640 1 -- 172.21.15.16:0/2511269406 wait complete. 2024-04-24T23:49:20.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-04-24T23:49:20.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-04-24T23:49:20.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-04-24T23:49:20.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-04-24T23:49:20.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-04-24T23:49:20.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-04-24T23:49:20.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-04-24T23:49:20.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-04-24T23:49:21.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.131+0000 7f05c51f6640 1 Processor -- start 2024-04-24T23:49:21.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.131+0000 7f05c51f6640 1 -- start start 2024-04-24T23:49:21.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.132+0000 7f05c51f6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f05c015ddf0 0x7f05c006bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.132+0000 7f05c51f6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05c00b2fd0 0x7f05c01618d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.132+0000 7f05c51f6640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05c015e1d0 con 0x7f05c0161e10 2024-04-24T23:49:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.132+0000 7f05c51f6640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f05c0162260 con 0x7f05c015ddf0 2024-04-24T23:49:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.132+0000 7f05c51f6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f05c006cb30 con 0x7f05c00b2fd0 2024-04-24T23:49:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.132+0000 7f05c49f5640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f05c0161e10 0x7f05c00cf470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50614/0 (socket says 172.21.15.16:50614) 2024-04-24T23:49:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.133+0000 7f05c49f5640 1 -- 172.21.15.16:0/1050565710 learned_addr learned my addr 172.21.15.16:0/1050565710 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.133+0000 7f05bf7fe640 1 --2- 172.21.15.16:0/1050565710 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05c00b2fd0 0x7f05c01618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.133+0000 7f05bf7fe640 1 -- 172.21.15.16:0/1050565710 >> v2:172.21.15.16:3300/0 conn(0x7f05c015ddf0 msgr2=0x7f05c006bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.133+0000 7f05bf7fe640 1 --2- 172.21.15.16:0/1050565710 >> v2:172.21.15.16:3300/0 conn(0x7f05c015ddf0 0x7f05c006bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.133+0000 7f05bf7fe640 1 -- 172.21.15.16:0/1050565710 >> v1:172.21.15.16:6789/0 conn(0x7f05c0161e10 legacy=0x7f05c00cf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.133+0000 7f05bf7fe640 1 -- 172.21.15.16:0/1050565710 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c006c570 con 0x7f05c00b2fd0 2024-04-24T23:49:21.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.134+0000 7f8fea010640 1 Processor -- start 2024-04-24T23:49:21.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.134+0000 7f8fea010640 1 -- start start 2024-04-24T23:49:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.133+0000 7f1fce058640 1 Processor -- start 2024-04-24T23:49:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.134+0000 7f4d4b177640 1 Processor -- start 2024-04-24T23:49:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.134+0000 7f4d4b177640 1 -- start start 2024-04-24T23:49:21.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f4d4b177640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d4415dff0 0x7f4d440cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f4d4b177640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d440b2a30 0x7f4d440d2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f4d4b177640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d440ced70 con 0x7f4d440d2d90 2024-04-24T23:49:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f4d4b177640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4d440cf050 con 0x7f4d4415dff0 2024-04-24T23:49:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f4d4b177640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f4d440cf330 con 0x7f4d440b2a30 2024-04-24T23:49:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f8fea010640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fdc0e3160 0x7f8fdc0e7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f8fea010640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fdc058810 con 0x7f8fdc053cb0 2024-04-24T23:49:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f8fea010640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fdc058af0 con 0x7f8fdc03c0a0 2024-04-24T23:49:21.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f8fea010640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8fdc007b70 con 0x7f8fdc0e3160 2024-04-24T23:49:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.133+0000 7f1fce058640 1 -- start start 2024-04-24T23:49:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f1fce058640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1fc003e830 0x7f1fc0053be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f1fce058640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc0058560 con 0x7f1fc00e30b0 2024-04-24T23:49:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f1fce058640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc0058840 con 0x7f1fc0054120 2024-04-24T23:49:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f1fce058640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f1fc0054590 con 0x7f1fc003e830 2024-04-24T23:49:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f1fc7fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f1fc0054120 0x7f1fc00068c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:41254/0 (socket says 172.21.15.16:41254) 2024-04-24T23:49:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f1fc7fff640 1 -- 172.21.15.16:0/1840352825 learned_addr learned my addr 172.21.15.16:0/1840352825 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f1fc6ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1fc003e830 0x7f1fc0053be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7f1fc67fc640 1 -- 172.21.15.16:0/1840352825 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2963088522 0 0) 0x7f1fc0058840 con 0x7f1fc0054120 2024-04-24T23:49:21.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7f1fc67fc640 1 -- 172.21.15.16:0/1840352825 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fb8003520 con 0x7f1fc0054120 2024-04-24T23:49:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.134+0000 7f05bf7fe640 1 --2- 172.21.15.16:0/1050565710 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05c00b2fd0 0x7f05c01618d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f05b00097a0 tx=0x7f05b002a0f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aed8a449e6b41bee server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.134+0000 7f05beffd640 1 -- 172.21.15.16:0/1050565710 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b0037070 con 0x7f05c00b2fd0 2024-04-24T23:49:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.134+0000 7f05beffd640 1 -- 172.21.15.16:0/1050565710 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05b002ae40 con 0x7f05c00b2fd0 2024-04-24T23:49:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f05c51f6640 1 -- 172.21.15.16:0/1050565710 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05c00b2fd0 msgr2=0x7f05c01618d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f05c51f6640 1 --2- 172.21.15.16:0/1050565710 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05c00b2fd0 0x7f05c01618d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f05b00097a0 tx=0x7f05b002a0f0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f05c51f6640 1 -- 172.21.15.16:0/1050565710 shutdown_connections 2024-04-24T23:49:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f05c51f6640 1 --2- 172.21.15.16:0/1050565710 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05c00b2fd0 0x7f05c01618d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f05c51f6640 1 --2- 172.21.15.16:0/1050565710 >> v2:172.21.15.16:3300/0 conn(0x7f05c015ddf0 0x7f05c006bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.135+0000 7f05c51f6640 1 -- 172.21.15.16:0/1050565710 >> 172.21.15.16:0/1050565710 conn(0x7f05c00d4060 msgr2=0x7f05c00d4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7f05c51f6640 1 -- 172.21.15.16:0/1050565710 shutdown_connections 2024-04-24T23:49:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7f1fc6ffd640 1 -- 172.21.15.16:0/1840352825 >> v1:172.21.15.156:6789/0 conn(0x7f1fc0054120 legacy=0x7f1fc00068c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7f1fc6ffd640 1 -- 172.21.15.16:0/1840352825 >> v1:172.21.15.16:6789/0 conn(0x7f1fc00e30b0 legacy=0x7f1fc0057d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:49:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7f1fc6ffd640 1 -- 172.21.15.16:0/1840352825 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc0058b20 con 0x7f1fc003e830 2024-04-24T23:49:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7f8fe880d640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f8fdc03c0a0 0x7f8fdc058050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:41278/0 (socket says 172.21.15.16:41278) 2024-04-24T23:49:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7f8fe880d640 1 -- 172.21.15.16:0/3369214637 learned_addr learned my addr 172.21.15.16:0/3369214637 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:21.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7f8fe900e640 1 --2- 172.21.15.16:0/3369214637 >> v2:172.21.15.16:3300/0 conn(0x7f8fdc0e3160 0x7f8fdc0e7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.137+0000 7f8fe900e640 1 -- 172.21.15.16:0/3369214637 >> v1:172.21.15.156:6789/0 conn(0x7f8fdc03c0a0 legacy=0x7f8fdc058050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7f4d49714640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f4d440d2d90 0x7f4d440b6f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:41270/0 (socket says 172.21.15.16:41270) 2024-04-24T23:49:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7f4d49714640 1 -- 172.21.15.16:0/4003516503 learned_addr learned my addr 172.21.15.16:0/4003516503 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7f4d48f13640 1 --2- 172.21.15.16:0/4003516503 >> v2:172.21.15.16:3300/0 conn(0x7f4d4415dff0 0x7f4d440cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7f4d43fff640 1 --2- 172.21.15.16:0/4003516503 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d440b2a30 0x7f4d440d2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.137+0000 7f4d437fe640 1 -- 172.21.15.16:0/4003516503 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3475927189 0 0) 0x7f4d440ced70 con 0x7f4d440d2d90 2024-04-24T23:49:21.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7f05c51f6640 1 -- 172.21.15.16:0/1050565710 wait complete. 2024-04-24T23:49:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7fad8d07d640 1 Processor -- start 2024-04-24T23:49:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.137+0000 7f8fe900e640 1 -- 172.21.15.16:0/3369214637 >> v1:172.21.15.16:6789/0 conn(0x7f8fdc053cb0 legacy=0x7f8fdc054100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.137+0000 7f8fe900e640 1 -- 172.21.15.16:0/3369214637 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fdc006360 con 0x7f8fdc0e3160 2024-04-24T23:49:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.137+0000 7f8fdbfff640 1 -- 172.21.15.16:0/3369214637 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3846782767 0 0) 0x7f8fdc058810 con 0x7f8fdc053cb0 2024-04-24T23:49:21.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.137+0000 7f4d437fe640 1 -- 172.21.15.16:0/4003516503 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d2c003520 con 0x7f4d440d2d90 2024-04-24T23:49:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.136+0000 7fad8d07d640 1 -- start start 2024-04-24T23:49:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.137+0000 7f1fc6ffd640 1 --2- 172.21.15.16:0/1840352825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fc003e830 0x7f1fc0053be0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1fb40097a0 tx=0x7f1fb402a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3269f956cd6149c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.137+0000 7f1fc67fc640 1 -- 172.21.15.16:0/1840352825 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4037070 con 0x7f1fc003e830 2024-04-24T23:49:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.137+0000 7f1fc67fc640 1 -- 172.21.15.16:0/1840352825 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fb4032440 con 0x7f1fc003e830 2024-04-24T23:49:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.137+0000 7f1fc67fc640 1 -- 172.21.15.16:0/1840352825 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb4032810 con 0x7f1fc003e830 2024-04-24T23:49:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7fad8d07d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad880b4ac0 0x7fad88142ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7fad8d07d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fad8815afb0 0x7fad8806b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7fad8d07d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad880cc230 con 0x7fad8815d850 2024-04-24T23:49:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7fad8d07d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fad88072470 con 0x7fad8815afb0 2024-04-24T23:49:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7fad8d07d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fad88071860 con 0x7fad880b4ac0 2024-04-24T23:49:21.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7f8e6b59e640 1 Processor -- start 2024-04-24T23:49:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.137+0000 7f4d43fff640 1 -- 172.21.15.16:0/4003516503 >> v2:172.21.15.16:3300/0 conn(0x7f4d4415dff0 msgr2=0x7f4d440cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.137+0000 7f4d43fff640 1 --2- 172.21.15.16:0/4003516503 >> v2:172.21.15.16:3300/0 conn(0x7f4d4415dff0 0x7f4d440cfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.137+0000 7f4d43fff640 1 -- 172.21.15.16:0/4003516503 >> v1:172.21.15.156:6789/0 conn(0x7f4d440d2d90 legacy=0x7f4d440b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.137+0000 7f4d43fff640 1 -- 172.21.15.16:0/4003516503 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d440cf610 con 0x7f4d440b2a30 2024-04-24T23:49:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7f4d48f13640 1 --2- 172.21.15.16:0/4003516503 >> v2:172.21.15.16:3300/0 conn(0x7f4d4415dff0 0x7f4d440cfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7f05c51f6640 1 Processor -- start 2024-04-24T23:49:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7fad86d9d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad880b4ac0 0x7fad88142ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7fad8759e640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fad8815d850 0x7fad8815dcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:41286/0 (socket says 172.21.15.16:41286) 2024-04-24T23:49:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7fad8759e640 1 -- 172.21.15.16:0/3012151092 learned_addr learned my addr 172.21.15.16:0/3012151092 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7f8e6b59e640 1 -- start start 2024-04-24T23:49:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7fad8659c640 1 --2- 172.21.15.16:0/3012151092 >> v2:172.21.15.16:3300/0 conn(0x7fad8815afb0 0x7fad8806b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7f05c51f6640 1 -- start start 2024-04-24T23:49:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7f1fce058640 1 -- 172.21.15.16:0/1840352825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fc003e830 msgr2=0x7f1fc0053be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7f1fce058640 1 --2- 172.21.15.16:0/1840352825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fc003e830 0x7f1fc0053be0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1fb40097a0 tx=0x7f1fb402a270 comp rx=0 tx=0).stop 2024-04-24T23:49:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7f1fce058640 1 -- 172.21.15.16:0/1840352825 shutdown_connections 2024-04-24T23:49:21.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7f8fe900e640 1 --2- 172.21.15.16:0/3369214637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fdc0e3160 0x7f8fdc0e7780 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4000b60 tx=0x7f8fe40a59d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da5c0cd2392edcd2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7f1fce058640 1 --2- 172.21.15.16:0/1840352825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fc003e830 0x7f1fc0053be0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.138+0000 7f1fce058640 1 -- 172.21.15.16:0/1840352825 >> 172.21.15.16:0/1840352825 conn(0x7f1fc00559d0 msgr2=0x7f1fc0055de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7f8fdbfff640 1 -- 172.21.15.16:0/3369214637 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe40af020 con 0x7f8fdc0e3160 2024-04-24T23:49:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7f8fdbfff640 1 -- 172.21.15.16:0/3369214637 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fe40ab950 con 0x7f8fdc0e3160 2024-04-24T23:49:21.148 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7f8fdbfff640 1 -- 172.21.15.16:0/3369214637 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe40af020 con 0x7f8fdc0e3160 2024-04-24T23:49:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7f1fce058640 1 -- 172.21.15.16:0/1840352825 shutdown_connections 2024-04-24T23:49:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7f1fce058640 1 -- 172.21.15.16:0/1840352825 wait complete. 2024-04-24T23:49:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7f05c51f6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05c015ddf0 0x7f05c0147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7f05c51f6640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05c0161e10 0x7f05c014a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7f05c51f6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05c014a780 0x7f05c00c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7f05c51f6640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f05c00d1ce0 con 0x7f05c014a780 2024-04-24T23:49:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7f05c51f6640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f05c00d2050 con 0x7f05c0161e10 2024-04-24T23:49:21.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7f05c51f6640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f05c00d34f0 con 0x7f05c015ddf0 2024-04-24T23:49:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7fad8659c640 1 -- 172.21.15.16:0/3012151092 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad880b4ac0 msgr2=0x7fad88142ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7fad8659c640 1 --2- 172.21.15.16:0/3012151092 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad880b4ac0 0x7fad88142ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7fad8659c640 1 -- 172.21.15.16:0/3012151092 >> v1:172.21.15.156:6789/0 conn(0x7fad8815d850 legacy=0x7fad8815dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.139+0000 7fad8659c640 1 -- 172.21.15.16:0/3012151092 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad88071b40 con 0x7fad8815afb0 2024-04-24T23:49:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f3ce4a58640 1 Processor -- start 2024-04-24T23:49:21.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f05bffff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05c015ddf0 0x7f05c0147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f05bffff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05c015ddf0 0x7f05c0147260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:41400/0 (socket says 172.21.15.16:41400) 2024-04-24T23:49:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f05bffff640 1 -- 172.21.15.16:0/2701474111 learned_addr learned my addr 172.21.15.16:0/2701474111 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f3ce4a58640 1 -- start start 2024-04-24T23:49:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8e6b59e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8e640df220 0x7f8e64038080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f4d43fff640 1 --2- 172.21.15.16:0/4003516503 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d440b2a30 0x7f4d440d2850 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009d90 tx=0x7f4d3c02a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dc1d830abaaa5e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8e6b59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e64053c70 0x7f8e6404fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8fea010640 1 -- 172.21.15.16:0/3369214637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fdc0e3160 msgr2=0x7f8fdc0e7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8fea010640 1 --2- 172.21.15.16:0/3369214637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fdc0e3160 0x7f8fdc0e7780 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4000b60 tx=0x7f8fe40a59d0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8e6b59e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8e64033ef0 0x7f8e6404ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8fea010640 1 -- 172.21.15.16:0/3369214637 shutdown_connections 2024-04-24T23:49:21.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8fea010640 1 --2- 172.21.15.16:0/3369214637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fdc0e3160 0x7f8fdc0e7780 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8fea010640 1 -- 172.21.15.16:0/3369214637 >> 172.21.15.16:0/3369214637 conn(0x7f8fdc058ee0 msgr2=0x7f8fdc0592c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f4d437fe640 1 -- 172.21.15.16:0/4003516503 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c007740 con 0x7f4d440b2a30 2024-04-24T23:49:21.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f4d437fe640 1 -- 172.21.15.16:0/4003516503 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d3c0078e0 con 0x7f4d440b2a30 2024-04-24T23:49:21.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f7b7c8f8640 1 Processor -- start 2024-04-24T23:49:21.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8fea010640 1 -- 172.21.15.16:0/3369214637 shutdown_connections 2024-04-24T23:49:21.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8e6b59e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8e640544c0 con 0x7f8e640df220 2024-04-24T23:49:21.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8e6ad9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8e64033ef0 0x7f8e6404ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f7b7c8f8640 1 -- start start 2024-04-24T23:49:21.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8e6ad9d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8e64033ef0 0x7f8e6404ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:34754/0 (socket says 172.21.15.16:34754) 2024-04-24T23:49:21.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8e6ad9d640 1 -- 172.21.15.16:0/3383462668 learned_addr learned my addr 172.21.15.16:0/3383462668 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8e6b59e640 1 -- 172.21.15.16:0/3383462668 --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8e640541e0 con 0x7f8e64033ef0 2024-04-24T23:49:21.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8e6b59e640 1 -- 172.21.15.16:0/3383462668 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8e64005f30 con 0x7f8e64053c70 2024-04-24T23:49:21.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8e69d9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e64053c70 0x7f8e6404fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f1fce058640 1 Processor -- start 2024-04-24T23:49:21.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.140+0000 7f8fea010640 1 -- 172.21.15.16:0/3369214637 wait complete. 2024-04-24T23:49:21.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f8e6ad9d640 1 -- 172.21.15.16:0/3383462668 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e64053c70 msgr2=0x7f8e6404fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f3ce4a58640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ce015a4e0 0x7f3ce00cde50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f8e6ad9d640 1 --2- 172.21.15.16:0/3383462668 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e64053c70 0x7f8e6404fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f3ce4a58640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3ce015afd0 0x7f3ce0163a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f3ce4a58640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ce00ce7c0 con 0x7f3ce006b620 2024-04-24T23:49:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f3ce4a58640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f3ce00ceaa0 con 0x7f3ce015a4e0 2024-04-24T23:49:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f3ce4a58640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f3ce00b2b30 con 0x7f3ce015afd0 2024-04-24T23:49:21.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f1fce058640 1 -- start start 2024-04-24T23:49:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f7b7c8f8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7b700065e0 0x7f7b70006a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f3cdffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3ce015afd0 0x7f3ce0163a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f3cdffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3ce015afd0 0x7f3ce0163a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:34768/0 (socket says 172.21.15.16:34768) 2024-04-24T23:49:21.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f3cdffff640 1 -- 172.21.15.16:0/1614525826 learned_addr learned my addr 172.21.15.16:0/1614525826 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f4d4b177640 1 -- 172.21.15.16:0/4003516503 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d440b2a30 msgr2=0x7f4d440d2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f4d4b177640 1 --2- 172.21.15.16:0/4003516503 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d440b2a30 0x7f4d440d2850 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c009d90 tx=0x7f4d3c02a660 comp rx=0 tx=0).stop 2024-04-24T23:49:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f7b7c8f8640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b7003d020 con 0x7f7b70006fb0 2024-04-24T23:49:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f7b7c8f8640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b70007b10 con 0x7f7b70006200 2024-04-24T23:49:21.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f7b7c8f8640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f7b70057a70 con 0x7f7b700065e0 2024-04-24T23:49:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f7b76ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7b700065e0 0x7f7b70006a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f05bffff640 1 -- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05c0161e10 msgr2=0x7f05c014a240 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:49:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f05bffff640 1 --2- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05c0161e10 0x7f05c014a240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f05bffff640 1 -- 172.21.15.16:0/2701474111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05c014a780 msgr2=0x7f05c00c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f05bffff640 1 --2- 172.21.15.16:0/2701474111 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05c014a780 0x7f05c00c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f05bffff640 1 -- 172.21.15.16:0/2701474111 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05c00c66c0 con 0x7f05c015ddf0 2024-04-24T23:49:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f7b777fe640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f7b70006200 0x7f7b70056fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:41292/0 (socket says 172.21.15.16:41292) 2024-04-24T23:49:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f7b777fe640 1 -- 172.21.15.16:0/1074727336 learned_addr learned my addr 172.21.15.16:0/1074727336 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.141+0000 7f8e6a59c640 1 --2- 172.21.15.16:0/3383462668 >> v2:172.21.15.16:3300/0 conn(0x7f8e640df220 0x7f8e64038080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f8e6ad9d640 1 -- 172.21.15.16:0/3383462668 >> v2:172.21.15.16:3300/0 conn(0x7f8e640df220 msgr2=0x7f8e64038080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f8e6ad9d640 1 --2- 172.21.15.16:0/3383462668 >> v2:172.21.15.16:3300/0 conn(0x7f8e640df220 0x7f8e64038080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f4d4b177640 1 -- 172.21.15.16:0/4003516503 shutdown_connections 2024-04-24T23:49:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f3cdf7fe640 1 --2- 172.21.15.16:0/1614525826 >> v2:172.21.15.16:3300/0 conn(0x7f3ce015a4e0 0x7f3ce00cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f4d4b177640 1 --2- 172.21.15.16:0/4003516503 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d440b2a30 0x7f4d440d2850 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f4d4b177640 1 --2- 172.21.15.16:0/4003516503 >> v2:172.21.15.16:3300/0 conn(0x7f4d4415dff0 0x7f4d440cfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f4d4b177640 1 -- 172.21.15.16:0/4003516503 >> 172.21.15.16:0/4003516503 conn(0x7f4d440d06f0 msgr2=0x7f4d440d0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f8e6ad9d640 1 -- 172.21.15.16:0/3383462668 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e64006210 con 0x7f8e64033ef0 2024-04-24T23:49:21.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f8e6a59c640 1 --2- 172.21.15.16:0/3383462668 >> v2:172.21.15.16:3300/0 conn(0x7f8e640df220 0x7f8e64038080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:21.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7fad8659c640 1 --2- 172.21.15.16:0/3012151092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad8815afb0 0x7fad8806b0e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fad780097a0 tx=0x7fad7802a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57eea7de8b21060 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f3cdeffd640 1 -- 172.21.15.16:0/1614525826 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4206986225 0 0) 0x7f3ce00ce7c0 con 0x7f3ce006b620 2024-04-24T23:49:21.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f3cdffff640 1 -- 172.21.15.16:0/1614525826 >> v2:172.21.15.16:3300/0 conn(0x7f3ce015a4e0 msgr2=0x7f3ce00cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f3cdffff640 1 --2- 172.21.15.16:0/1614525826 >> v2:172.21.15.16:3300/0 conn(0x7f3ce015a4e0 0x7f3ce00cde50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f3cdffff640 1 -- 172.21.15.16:0/1614525826 >> v1:172.21.15.156:6789/0 conn(0x7f3ce006b620 legacy=0x7f3ce015a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f3cdffff640 1 -- 172.21.15.16:0/1614525826 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce00b2e10 con 0x7f3ce015afd0 2024-04-24T23:49:21.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f8fea010640 1 Processor -- start 2024-04-24T23:49:21.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7fad85d9b640 1 -- 172.21.15.16:0/3012151092 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78037070 con 0x7fad8815afb0 2024-04-24T23:49:21.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f8fea010640 1 -- start start 2024-04-24T23:49:21.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 797491239 0 0) 0x7f7b7003d020 con 0x7f7b70006fb0 2024-04-24T23:49:21.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b68003520 con 0x7f7b70006fb0 2024-04-24T23:49:21.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7fad85d9b640 1 -- 172.21.15.16:0/3012151092 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad7802ad10 con 0x7fad8815afb0 2024-04-24T23:49:21.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f1fce058640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc003e830 0x7f1fc00d5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7fad85d9b640 1 -- 172.21.15.16:0/3012151092 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad78032660 con 0x7fad8815afb0 2024-04-24T23:49:21.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3153124126 0 0) 0x7f7b70007b10 con 0x7f7b70006200 2024-04-24T23:49:21.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7b7003d020 con 0x7f7b70006200 2024-04-24T23:49:21.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f1fc77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc003e830 0x7f1fc00d5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f05bffff640 1 --2- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05c015ddf0 0x7f05c0147260 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f05b800ba80 tx=0x7f05b800bf40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 781151188 0 0) 0x7f7b68003520 con 0x7f7b70006fb0 2024-04-24T23:49:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 >> v2:172.21.15.156:3300/0 conn(0x7f7b700065e0 msgr2=0x7f7b70006a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 --2- 172.21.15.16:0/1074727336 >> v2:172.21.15.156:3300/0 conn(0x7f7b700065e0 0x7f7b70006a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 >> v1:172.21.15.156:6789/0 conn(0x7f7b70006200 legacy=0x7f7b70056fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b70057d50 con 0x7f7b70006fb0 2024-04-24T23:49:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f7b64001670 con 0x7f7b70006fb0 2024-04-24T23:49:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 >> v1:172.21.15.16:6789/0 conn(0x7f7b70006fb0 legacy=0x7f7b70007400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 --2- 172.21.15.16:0/1074727336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7b68003b40 0x7f7b68003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 --2- 172.21.15.16:0/1074727336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b68004800 0x7f7b68004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 --2- 172.21.15.16:0/1074727336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b680051c0 0x7f7b6800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b70057a70 con 0x7f7b680051c0 2024-04-24T23:49:21.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6800bf30 con 0x7f7b68004800 2024-04-24T23:49:21.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7b6800c130 con 0x7f7b68003b40 2024-04-24T23:49:21.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f05bcff9640 1 -- 172.21.15.16:0/2701474111 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b8019070 con 0x7f05c015ddf0 2024-04-24T23:49:21.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f1fc77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc003e830 0x7f1fc00d5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48374/0 (socket says 172.21.15.16:48374) 2024-04-24T23:49:21.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f05c51f6640 1 -- 172.21.15.16:0/2701474111 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05c00c69a0 con 0x7f05c015ddf0 2024-04-24T23:49:21.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f1fc77fe640 1 -- 172.21.15.16:0/1568975994 learned_addr learned my addr 172.21.15.16:0/1568975994 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b77fff640 1 --2- 172.21.15.16:0/1074727336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b680051c0 0x7f7b6800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f7b76ffd640 1 --2- 172.21.15.16:0/1074727336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b68004800 0x7f7b68004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.142+0000 7f4d4b177640 1 -- 172.21.15.16:0/4003516503 shutdown_connections 2024-04-24T23:49:21.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f1fce058640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fc0054120 0x7f1fc00da900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f05c51f6640 1 -- 172.21.15.16:0/2701474111 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f05c00c7cd0 con 0x7f05c015ddf0 2024-04-24T23:49:21.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f1fce058640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fc00e30b0 0x7f1fc00dae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7fad8d07d640 1 -- 172.21.15.16:0/3012151092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad8815afb0 msgr2=0x7fad8806b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7fad8d07d640 1 --2- 172.21.15.16:0/3012151092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad8815afb0 0x7fad8806b0e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fad780097a0 tx=0x7fad7802a080 comp rx=0 tx=0).stop 2024-04-24T23:49:21.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f1fce058640 1 -- 172.21.15.16:0/1568975994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc00e3490 con 0x7f1fc003e830 2024-04-24T23:49:21.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f1fce058640 1 -- 172.21.15.16:0/1568975994 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc003bfb0 con 0x7f1fc0054120 2024-04-24T23:49:21.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7fad8d07d640 1 -- 172.21.15.16:0/3012151092 shutdown_connections 2024-04-24T23:49:21.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7fad8d07d640 1 --2- 172.21.15.16:0/3012151092 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad8815afb0 0x7fad8806b0e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7fad8d07d640 1 --2- 172.21.15.16:0/3012151092 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad880b4ac0 0x7fad88142ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7fad8d07d640 1 -- 172.21.15.16:0/3012151092 >> 172.21.15.16:0/3012151092 conn(0x7fad880af800 msgr2=0x7fad880afc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f1fce058640 1 -- 172.21.15.16:0/1568975994 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1fc0006fd0 con 0x7f1fc00e30b0 2024-04-24T23:49:21.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7fad8d07d640 1 -- 172.21.15.16:0/3012151092 shutdown_connections 2024-04-24T23:49:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f1fc7fff640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fc00e30b0 0x7f1fc00dae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f1fc6ffd640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fc0054120 0x7f1fc00da900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7fad8d07d640 1 -- 172.21.15.16:0/3012151092 wait complete. 2024-04-24T23:49:21.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f8fea010640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fdc03c0a0 0x7f8fdc052a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f8fea010640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fdc053cb0 0x7f8fdc039c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f8fea010640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fdc0e3160 0x7f8fdc180b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f7b77fff640 1 -- 172.21.15.16:0/1074727336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7b68003b40 msgr2=0x7f7b68003f20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:49:21.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f8fea010640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8fdc0ee050 con 0x7f8fdc0e3160 2024-04-24T23:49:21.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f8fea010640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8fdc0e3540 con 0x7f8fdc053cb0 2024-04-24T23:49:21.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f8fea010640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8fdc0ee1d0 con 0x7f8fdc03c0a0 2024-04-24T23:49:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f7b77fff640 1 --2- 172.21.15.16:0/1074727336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7b68003b40 0x7f7b68003f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f7b77fff640 1 -- 172.21.15.16:0/1074727336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b68004800 msgr2=0x7f7b68004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f7b77fff640 1 --2- 172.21.15.16:0/1074727336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b68004800 0x7f7b68004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f1fc77fe640 1 -- 172.21.15.16:0/1568975994 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fc00e30b0 msgr2=0x7f1fc00dae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f1fc77fe640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1fc00e30b0 0x7f1fc00dae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f1fc77fe640 1 -- 172.21.15.16:0/1568975994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fc0054120 msgr2=0x7f1fc00da900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f8fe900e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fdc03c0a0 0x7f8fdc052a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f8fe900e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fdc03c0a0 0x7f8fdc052a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:41442/0 (socket says 172.21.15.16:41442) 2024-04-24T23:49:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f8fe900e640 1 -- 172.21.15.16:0/2721598489 learned_addr learned my addr 172.21.15.16:0/2721598489 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f8fe980f640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fdc0e3160 0x7f8fdc180b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f1fc77fe640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1fc0054120 0x7f1fc00da900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f8fe980f640 1 -- 172.21.15.16:0/2721598489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fdc03c0a0 msgr2=0x7f8fdc052a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f8fe980f640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fdc03c0a0 0x7f8fdc052a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f8fe980f640 1 -- 172.21.15.16:0/2721598489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fdc053cb0 msgr2=0x7f8fdc039c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f8fe880d640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fdc053cb0 0x7f8fdc039c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f1fc77fe640 1 -- 172.21.15.16:0/1568975994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc00db950 con 0x7f1fc003e830 2024-04-24T23:49:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f1fc77fe640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc003e830 0x7f1fc00d5f50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00b780 tx=0x7f1fbc00bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f8fe980f640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fdc053cb0 0x7f8fdc039c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f8fe980f640 1 -- 172.21.15.16:0/2721598489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fdc03dc00 con 0x7f8fdc0e3160 2024-04-24T23:49:21.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f8fe880d640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fdc053cb0 0x7f8fdc039c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:21.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.144+0000 7f7b77fff640 1 -- 172.21.15.16:0/1074727336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b68003520 con 0x7f7b680051c0 2024-04-24T23:49:21.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f05a67fc640 1 -- 172.21.15.16:0/2701474111 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f05a0001030 con 0x7f05c015ddf0 2024-04-24T23:49:21.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f1fb3fff640 1 -- 172.21.15.16:0/1568975994 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc019070 con 0x7f1fc003e830 2024-04-24T23:49:21.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f7b77fff640 1 -- 172.21.15.16:0/1074727336 reap_dead start 2024-04-24T23:49:21.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f1fb3fff640 1 -- 172.21.15.16:0/1568975994 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fbc00c720 con 0x7f1fc003e830 2024-04-24T23:49:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.143+0000 7f4d4b177640 1 -- 172.21.15.16:0/4003516503 wait complete. 2024-04-24T23:49:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f4d4b177640 1 Processor -- start 2024-04-24T23:49:21.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f8fe980f640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fdc0e3160 0x7f8fdc180b40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000c440 tx=0x7f8fe000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f1fb3fff640 1 -- 172.21.15.16:0/1568975994 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc00cb10 con 0x7f1fc003e830 2024-04-24T23:49:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f1fce058640 1 -- 172.21.15.16:0/1568975994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc00dbc30 con 0x7f1fc003e830 2024-04-24T23:49:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f7b77fff640 1 --2- 172.21.15.16:0/1074727336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b680051c0 0x7f7b6800b9c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7b64007f80 tx=0x7f7b68005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773b421fc9f93fcc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f8fd9ffb640 1 -- 172.21.15.16:0/2721598489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe0017070 con 0x7f8fdc0e3160 2024-04-24T23:49:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f8fd9ffb640 1 -- 172.21.15.16:0/2721598489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fe00035f0 con 0x7f8fdc0e3160 2024-04-24T23:49:21.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.146+0000 7f8fd9ffb640 1 -- 172.21.15.16:0/2721598489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fe00039e0 con 0x7f8fdc0e3160 2024-04-24T23:49:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b64007bd0 con 0x7f7b680051c0 2024-04-24T23:49:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.146+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b64031620 con 0x7f7b680051c0 2024-04-24T23:49:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.146+0000 7f7b767fc640 1 -- 172.21.15.16:0/1074727336 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b64031a10 con 0x7f7b680051c0 2024-04-24T23:49:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.146+0000 7fad8d07d640 1 Processor -- start 2024-04-24T23:49:21.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.146+0000 7f1fce058640 1 -- 172.21.15.16:0/1568975994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc00dc1f0 con 0x7f1fc003e830 2024-04-24T23:49:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.146+0000 7fad8d07d640 1 -- start start 2024-04-24T23:49:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.146+0000 7f8fea010640 1 -- 172.21.15.16:0/2721598489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fdc03dee0 con 0x7f8fdc0e3160 2024-04-24T23:49:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f05bcff9640 1 -- 172.21.15.16:0/2701474111 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05b800cc20 con 0x7f05c015ddf0 2024-04-24T23:49:21.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.146+0000 7f8fea010640 1 -- 172.21.15.16:0/2721598489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fdc03e4a0 con 0x7f8fdc0e3160 2024-04-24T23:49:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.146+0000 7f7b7c8f8640 1 -- 172.21.15.16:0/1074727336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b680051c0 msgr2=0x7f7b6800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.146+0000 7f7b7c8f8640 1 --2- 172.21.15.16:0/1074727336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b680051c0 0x7f7b6800b9c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7b64007f80 tx=0x7f7b68005830 comp rx=0 tx=0).stop 2024-04-24T23:49:21.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f4d4b177640 1 -- start start 2024-04-24T23:49:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f7b7c8f8640 1 -- 172.21.15.16:0/1074727336 shutdown_connections 2024-04-24T23:49:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f7b7c8f8640 1 --2- 172.21.15.16:0/1074727336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b680051c0 0x7f7b6800b9c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f7b7c8f8640 1 -- 172.21.15.16:0/1074727336 >> 172.21.15.16:0/1074727336 conn(0x7f7b70058ee0 msgr2=0x7f7b700592f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.145+0000 7f8e6ad9d640 1 --2- 172.21.15.16:0/3383462668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e64033ef0 0x7f8e6404ffd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8e64033c60 tx=0x7f8e5c02a140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83d43d3d36b23d66 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.146+0000 7f8e6959a640 1 -- 172.21.15.16:0/3383462668 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c037070 con 0x7f8e64033ef0 2024-04-24T23:49:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.146+0000 7f8e6959a640 1 -- 172.21.15.16:0/3383462668 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e5c032440 con 0x7f8e64033ef0 2024-04-24T23:49:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.146+0000 7f8e6959a640 1 -- 172.21.15.16:0/3383462668 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c032810 con 0x7f8e64033ef0 2024-04-24T23:49:21.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f4d4b177640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d440b2a30 0x7f4d441f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f4d4b177640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d440d2d90 0x7f4d440c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f4d4b177640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d4415dff0 0x7f4d440c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f4d4b177640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4406c860 con 0x7f4d440b2a30 2024-04-24T23:49:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f4d4b177640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4406c6e0 con 0x7f4d4415dff0 2024-04-24T23:49:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f4d4b177640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f4d4406c9e0 con 0x7f4d440d2d90 2024-04-24T23:49:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f7b7c8f8640 1 -- 172.21.15.16:0/1074727336 shutdown_connections 2024-04-24T23:49:21.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f4d48f13640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d440b2a30 0x7f4d441f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f05bcff9640 1 -- 172.21.15.16:0/2701474111 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05b8014440 con 0x7f05c015ddf0 2024-04-24T23:49:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7fad8d07d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad880b4ac0 0x7fad8813a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f05bcff9640 1 -- 172.21.15.16:0/2701474111 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f05b801c070 con 0x7f05c015ddf0 2024-04-24T23:49:21.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f4d48f13640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d440b2a30 0x7f4d441f1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48384/0 (socket says 172.21.15.16:48384) 2024-04-24T23:49:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f4d48f13640 1 -- 172.21.15.16:0/2448188098 learned_addr learned my addr 172.21.15.16:0/2448188098 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f7b7c8f8640 1 -- 172.21.15.16:0/1074727336 wait complete. 2024-04-24T23:49:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7fad8d07d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad8815d850 0x7fad8813da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f1fce058640 1 -- 172.21.15.16:0/1568975994 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1fc00057d0 con 0x7f1fc003e830 2024-04-24T23:49:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7f3cdffff640 1 --2- 172.21.15.16:0/1614525826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce015afd0 0x7f3ce0163a90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3ce006c5e0 tx=0x7f3cd802a290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a78c5ea04301f34d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7fad8d07d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad8813dfb0 0x7fad881582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7fad8d07d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fad8806b620 con 0x7fad8813dfb0 2024-04-24T23:49:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7fad8d07d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fad88072100 con 0x7fad880b4ac0 2024-04-24T23:49:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7fad8d07d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fad8806b7a0 con 0x7fad8815d850 2024-04-24T23:49:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7fad86d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad880b4ac0 0x7fad8813a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7fad86d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad880b4ac0 0x7fad8813a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:34838/0 (socket says 172.21.15.16:34838) 2024-04-24T23:49:21.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7fad86d9d640 1 -- 172.21.15.16:0/2799282109 learned_addr learned my addr 172.21.15.16:0/2799282109 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7f8fea010640 1 -- 172.21.15.16:0/2721598489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8fdc005390 con 0x7f8fdc0e3160 2024-04-24T23:49:21.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7fad8659c640 1 --2- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad8815d850 0x7fad8813da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f8e6b59e640 1 -- 172.21.15.16:0/3383462668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e64033ef0 msgr2=0x7f8e6404ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f8e6b59e640 1 --2- 172.21.15.16:0/3383462668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e64033ef0 0x7f8e6404ffd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8e64033c60 tx=0x7f8e5c02a140 comp rx=0 tx=0).stop 2024-04-24T23:49:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f1fb3fff640 1 -- 172.21.15.16:0/1568975994 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1fbc01c070 con 0x7f1fc003e830 2024-04-24T23:49:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7f3cdeffd640 1 -- 172.21.15.16:0/1614525826 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8037070 con 0x7f3ce015afd0 2024-04-24T23:49:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7f3cdeffd640 1 -- 172.21.15.16:0/1614525826 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cd8032440 con 0x7f3ce015afd0 2024-04-24T23:49:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f4d43fff640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d440d2d90 0x7f4d440c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.147+0000 7f4d49714640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d4415dff0 0x7f4d440c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7f4d49714640 1 -- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d440d2d90 msgr2=0x7f4d440c99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7f4d49714640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d440d2d90 0x7f4d440c99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7f4d49714640 1 -- 172.21.15.16:0/2448188098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d440b2a30 msgr2=0x7f4d441f1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f7b7c8f8640 1 Processor -- start 2024-04-24T23:49:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f05bcff9640 1 --2- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f05ac06bf60 0x7f05ac06e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7fad86d9d640 1 -- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad8815d850 msgr2=0x7fad8813da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7fad86d9d640 1 --2- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fad8815d850 0x7fad8813da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7fad86d9d640 1 -- 172.21.15.16:0/2799282109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad8813dfb0 msgr2=0x7fad881582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7fad86d9d640 1 --2- 172.21.15.16:0/2799282109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad8813dfb0 0x7fad881582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.148+0000 7fad86d9d640 1 -- 172.21.15.16:0/2799282109 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad88157280 con 0x7fad880b4ac0 2024-04-24T23:49:21.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7fad86d9d640 1 --2- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad880b4ac0 0x7fad8813a320 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fad8000ce40 tx=0x7fad8000a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f3cdeffd640 1 -- 172.21.15.16:0/1614525826 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cd8032830 con 0x7f3ce015afd0 2024-04-24T23:49:21.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f05bcff9640 1 -- 172.21.15.16:0/2701474111 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f05b808b0b0 con 0x7f05c015ddf0 2024-04-24T23:49:21.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f05bf7fe640 1 --2- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f05ac06bf60 0x7f05ac06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f7b7c8f8640 1 -- start start 2024-04-24T23:49:21.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f8e6b59e640 1 -- 172.21.15.16:0/3383462668 shutdown_connections 2024-04-24T23:49:21.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f8e6b59e640 1 --2- 172.21.15.16:0/3383462668 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e64033ef0 0x7f8e6404ffd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f8e6b59e640 1 --2- 172.21.15.16:0/3383462668 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e64053c70 0x7f8e6404fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f8e6b59e640 1 --2- 172.21.15.16:0/3383462668 >> v2:172.21.15.16:3300/0 conn(0x7f8e640df220 0x7f8e64038080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f05bcff9640 1 --2- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:6816/667041555,v1:172.21.15.156:6817/667041555] conn(0x7f05ac0756d0 0x7f05ac077b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f05bcff9640 1 -- 172.21.15.16:0/2701474111 --> [v2:172.21.15.156:6816/667041555,v1:172.21.15.156:6817/667041555] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05b800cc20 con 0x7f05ac0756d0 2024-04-24T23:49:21.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f8e6b59e640 1 -- 172.21.15.16:0/3383462668 >> 172.21.15.16:0/3383462668 conn(0x7f8e64009fb0 msgr2=0x7f8e6400a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f05bcff9640 1 -- 172.21.15.16:0/2701474111 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f05b8092050 con 0x7f05c015ddf0 2024-04-24T23:49:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f1fb3fff640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1fac06beb0 0x7f1fac06e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f3ce4a58640 1 -- 172.21.15.16:0/1614525826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce015afd0 msgr2=0x7f3ce0163a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f3ce4a58640 1 --2- 172.21.15.16:0/1614525826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce015afd0 0x7f3ce0163a90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3ce006c5e0 tx=0x7f3cd802a290 comp rx=0 tx=0).stop 2024-04-24T23:49:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f1fb3fff640 1 -- 172.21.15.16:0/1568975994 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1fbc08a640 con 0x7f1fc003e830 2024-04-24T23:49:21.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f1fc6ffd640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1fac06beb0 0x7f1fac06e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7fad6f7fe640 1 -- 172.21.15.16:0/2799282109 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad80003bc0 con 0x7fad880b4ac0 2024-04-24T23:49:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f8e6b59e640 1 -- 172.21.15.16:0/3383462668 shutdown_connections 2024-04-24T23:49:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f1fb3fff640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.16:6802/4175732037,v1:172.21.15.16:6804/4175732037] conn(0x7f1fac075630 0x7f1fac077a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f3ce4a58640 1 -- 172.21.15.16:0/1614525826 shutdown_connections 2024-04-24T23:49:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f3ce4a58640 1 --2- 172.21.15.16:0/1614525826 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce015afd0 0x7f3ce0163a90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f3ce4a58640 1 --2- 172.21.15.16:0/1614525826 >> v2:172.21.15.16:3300/0 conn(0x7f3ce015a4e0 0x7f3ce00cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7f3ce4a58640 1 -- 172.21.15.16:0/1614525826 >> 172.21.15.16:0/1614525826 conn(0x7f3ce00af700 msgr2=0x7f3ce00afb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7fad6f7fe640 1 -- 172.21.15.16:0/2799282109 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad80003d60 con 0x7fad880b4ac0 2024-04-24T23:49:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.149+0000 7fad6f7fe640 1 -- 172.21.15.16:0/2799282109 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad80004240 con 0x7fad880b4ac0 2024-04-24T23:49:21.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f3ce4a58640 1 -- 172.21.15.16:0/1614525826 shutdown_connections 2024-04-24T23:49:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7fad8d07d640 1 -- 172.21.15.16:0/2799282109 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad88157560 con 0x7fad880b4ac0 2024-04-24T23:49:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f1fc7fff640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.16:6802/4175732037,v1:172.21.15.16:6804/4175732037] conn(0x7f1fac075630 0x7f1fac077a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f8e6b59e640 1 -- 172.21.15.16:0/3383462668 wait complete. 2024-04-24T23:49:21.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7fad8d07d640 1 -- 172.21.15.16:0/2799282109 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad881427f0 con 0x7fad880b4ac0 2024-04-24T23:49:21.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f4d49714640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d440b2a30 0x7f4d441f1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f4d49714640 1 -- 172.21.15.16:0/2448188098 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d440caa40 con 0x7f4d4415dff0 2024-04-24T23:49:21.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f1fb3fff640 1 -- 172.21.15.16:0/1568975994 --> [v2:172.21.15.16:6802/4175732037,v1:172.21.15.16:6804/4175732037] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1fbc00ce50 con 0x7f1fac075630 2024-04-24T23:49:21.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f4d43fff640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4d440d2d90 0x7f4d440c99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:21.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f4d48f13640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d440b2a30 0x7f4d441f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:21.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f1fb3fff640 1 -- 172.21.15.16:0/1568975994 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1fbc091050 con 0x7f1fc003e830 2024-04-24T23:49:21.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f05c49f5640 1 --2- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:6816/667041555,v1:172.21.15.156:6817/667041555] conn(0x7f05ac0756d0 0x7f05ac077b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f1fc6ffd640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1fac06beb0 0x7f1fac06e370 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4034000 tx=0x7f1fb402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f7b7c8f8640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7b680051c0 0x7f7b70038840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f05bf7fe640 1 --2- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f05ac06bf60 0x7f05ac06e420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f05b0034000 tx=0x7f05b002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f7b7c8f8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b7004f310 0x7f7b7003b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f8fd9ffb640 1 -- 172.21.15.16:0/2721598489 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8fe001a070 con 0x7f8fdc0e3160 2024-04-24T23:49:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f8fd9ffb640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8fc806bf00 0x7f8fc806e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f3ce4a58640 1 -- 172.21.15.16:0/1614525826 wait complete. 2024-04-24T23:49:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.150+0000 7f4d49714640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d4415dff0 0x7f4d440c77a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800c3f0 tx=0x7f4d3800c8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f7b7c8f8640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b700e47e0 0x7f7b7004e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f8fe900e640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8fc806bf00 0x7f8fc806e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f8fd9ffb640 1 -- 172.21.15.16:0/2721598489 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8fe008ab30 con 0x7f8fdc0e3160 2024-04-24T23:49:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f4d417fa640 1 -- 172.21.15.16:0/2448188098 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d38017070 con 0x7f4d4415dff0 2024-04-24T23:49:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f4d417fa640 1 -- 172.21.15.16:0/2448188098 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d38005ce0 con 0x7f4d4415dff0 2024-04-24T23:49:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f4d417fa640 1 -- 172.21.15.16:0/2448188098 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d380072f0 con 0x7f4d4415dff0 2024-04-24T23:49:21.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f05c49f5640 1 --2- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:6816/667041555,v1:172.21.15.156:6817/667041555] conn(0x7f05ac0756d0 0x7f05ac077b10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f7b76ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b7004f310 0x7f7b7003b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f8fd9ffb640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.16:6800/2543873173,v1:172.21.15.16:6801/2543873173] conn(0x7f8fc8075680 0x7f8fc8077ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f4d4b177640 1 -- 172.21.15.16:0/2448188098 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d441f2100 con 0x7f4d4415dff0 2024-04-24T23:49:21.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f7b76ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b7004f310 0x7f7b7003b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48406/0 (socket says 172.21.15.16:48406) 2024-04-24T23:49:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f4d4b177640 1 -- 172.21.15.16:0/2448188098 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d440ca760 con 0x7f4d4415dff0 2024-04-24T23:49:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f7b76ffd640 1 -- 172.21.15.16:0/1235783943 learned_addr learned my addr 172.21.15.16:0/1235783943 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f7b7c8f8640 1 -- 172.21.15.16:0/1235783943 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7b700585f0 con 0x7f7b7004f310 2024-04-24T23:49:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f7b7c8f8640 1 -- 172.21.15.16:0/1235783943 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7b70058310 con 0x7f7b700e47e0 2024-04-24T23:49:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f7b77fff640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b700e47e0 0x7f7b7004e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f7b7c8f8640 1 -- 172.21.15.16:0/1235783943 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7b700e3520 con 0x7f7b680051c0 2024-04-24T23:49:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f8fe880d640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.16:6800/2543873173,v1:172.21.15.16:6801/2543873173] conn(0x7f8fc8075680 0x7f8fc8077ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f8fd9ffb640 1 -- 172.21.15.16:0/2721598489 --> [v2:172.21.15.16:6800/2543873173,v1:172.21.15.16:6801/2543873173] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8fc8078190 con 0x7f8fc8075680 2024-04-24T23:49:21.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f8fd9ffb640 1 -- 172.21.15.16:0/2721598489 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8fe008af20 con 0x7f8fdc0e3160 2024-04-24T23:49:21.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7fad8d07d640 1 -- 172.21.15.16:0/2799282109 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fad8804f500 con 0x7fad880b4ac0 2024-04-24T23:49:21.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f8e6b59e640 1 Processor -- start 2024-04-24T23:49:21.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f8fe880d640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.16:6800/2543873173,v1:172.21.15.16:6801/2543873173] conn(0x7f8fc8075680 0x7f8fc8077ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.151+0000 7f1fc7fff640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.16:6802/4175732037,v1:172.21.15.16:6804/4175732037] conn(0x7f1fac075630 0x7f1fac077a70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f8e6b59e640 1 -- start start 2024-04-24T23:49:21.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f1fb3fff640 1 -- 172.21.15.16:0/1568975994 <== osd.2 v2:172.21.15.16:6802/4175732037 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1fbc00ce50 con 0x7f1fac075630 2024-04-24T23:49:21.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f3ce4a58640 1 Processor -- start 2024-04-24T23:49:21.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f7b777fe640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7b680051c0 0x7f7b70038840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f7b777fe640 1 -- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b700e47e0 msgr2=0x7f7b7004e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f7b777fe640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b700e47e0 0x7f7b7004e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f7b777fe640 1 -- 172.21.15.16:0/1235783943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b7004f310 msgr2=0x7f7b7003b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f7b777fe640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b7004f310 0x7f7b7003b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f7b777fe640 1 -- 172.21.15.16:0/1235783943 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b70037a40 con 0x7f7b680051c0 2024-04-24T23:49:21.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.152+0000 7f8fe900e640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8fc806bf00 0x7f8fc806e3c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8fe402d700 tx=0x7f8fe40ab000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.153+0000 7f4d4b177640 1 -- 172.21.15.16:0/2448188098 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4d24001030 con 0x7f4d4415dff0 2024-04-24T23:49:21.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.153+0000 7f7b76ffd640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7b7004f310 0x7f7b7003b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:21.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.153+0000 7f7b777fe640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7b680051c0 0x7f7b70038840 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7b78050ce0 tx=0x7f7b78068530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.153+0000 7f7b77fff640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7b700e47e0 0x7f7b7004e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:21.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.153+0000 7f3ce4a58640 1 -- start start 2024-04-24T23:49:21.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.153+0000 7f7b63fff640 1 -- 172.21.15.16:0/1235783943 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7806f070 con 0x7f7b680051c0 2024-04-24T23:49:21.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.153+0000 7f7b63fff640 1 -- 172.21.15.16:0/1235783943 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b7806d030 con 0x7f7b680051c0 2024-04-24T23:49:21.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.153+0000 7f7b63fff640 1 -- 172.21.15.16:0/1235783943 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78094a50 con 0x7f7b680051c0 2024-04-24T23:49:21.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.153+0000 7f05bcff9640 1 -- 172.21.15.16:0/2701474111 <== osd.6 v2:172.21.15.156:6816/667041555 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f05b800cc20 con 0x7f05ac0756d0 2024-04-24T23:49:21.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.153+0000 7f7b7c8f8640 1 -- 172.21.15.16:0/1235783943 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b7004d940 con 0x7f7b680051c0 2024-04-24T23:49:21.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.153+0000 7f7b7c8f8640 1 -- 172.21.15.16:0/1235783943 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b7004df00 con 0x7f7b680051c0 2024-04-24T23:49:21.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.153+0000 7fad6f7fe640 1 -- 172.21.15.16:0/2799282109 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fad800043e0 con 0x7fad880b4ac0 2024-04-24T23:49:21.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f8e6b59e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e64033ef0 0x7f8e64175570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7fad6f7fe640 1 --2- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fad7406bf00 0x7fad7406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f8e6b59e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e64053c70 0x7f8e640498a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f8e6b59e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e640df220 0x7f8e640d18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7fad6f7fe640 1 -- 172.21.15.16:0/2799282109 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fad8008bbb0 con 0x7fad880b4ac0 2024-04-24T23:49:21.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f8e6b59e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8e64037a90 con 0x7f8e64033ef0 2024-04-24T23:49:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f8e6b59e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8e64037910 con 0x7f8e64053c70 2024-04-24T23:49:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f8e6b59e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8e640df600 con 0x7f8e640df220 2024-04-24T23:49:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f8e6a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e64033ef0 0x7f8e64175570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f3ce4a58640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce006b620 0x7f3ce01660d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f3ce4a58640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ce015a4e0 0x7f3ce013edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f3ce4a58640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3ce015afd0 0x7f3ce013f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f8e6a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e64033ef0 0x7f8e64175570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48414/0 (socket says 172.21.15.16:48414) 2024-04-24T23:49:21.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f8e6a59c640 1 -- 172.21.15.16:0/1129702946 learned_addr learned my addr 172.21.15.16:0/1129702946 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7fad6f7fe640 1 --2- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:6824/2447858915,v1:172.21.15.156:6825/2447858915] conn(0x7fad74075670 0x7fad74077ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7fad6f7fe640 1 -- 172.21.15.16:0/2799282109 --> [v2:172.21.15.156:6824/2447858915,v1:172.21.15.156:6825/2447858915] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fad74078180 con 0x7fad74075670 2024-04-24T23:49:21.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7fad6f7fe640 1 -- 172.21.15.16:0/2799282109 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fad800902a0 con 0x7fad880b4ac0 2024-04-24T23:49:21.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f3ce4a58640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00ce390 con 0x7f3ce015a4e0 2024-04-24T23:49:21.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f3ce4a58640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3ce00ced80 con 0x7f3ce006b620 2024-04-24T23:49:21.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f3ce4a58640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f3ce00ce510 con 0x7f3ce015afd0 2024-04-24T23:49:21.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.154+0000 7f4d417fa640 1 -- 172.21.15.16:0/2448188098 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4d3801a070 con 0x7f4d4415dff0 2024-04-24T23:49:21.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f7b7c8f8640 1 -- 172.21.15.16:0/1235783943 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7b700057e0 con 0x7f7b680051c0 2024-04-24T23:49:21.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f8e69d9b640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e64053c70 0x7f8e640498a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7fad8659c640 1 --2- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fad7406bf00 0x7fad7406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f8e6ad9d640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e640df220 0x7f8e640d18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f8e6a59c640 1 -- 172.21.15.16:0/1129702946 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e640df220 msgr2=0x7f8e640d18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f8fd9ffb640 1 -- 172.21.15.16:0/2721598489 <== osd.1 v2:172.21.15.16:6800/2543873173 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8fc8078190 con 0x7f8fc8075680 2024-04-24T23:49:21.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f3cdffff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3ce015afd0 0x7f3ce013f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7fad8759e640 1 --2- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:6824/2447858915,v1:172.21.15.156:6825/2447858915] conn(0x7fad74075670 0x7fad74077ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f4d417fa640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4d3006bf00 0x7f4d3006e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f3cd6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ce015a4e0 0x7f3ce013edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f4d417fa640 1 -- 172.21.15.16:0/2448188098 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f4d3808a770 con 0x7f4d4415dff0 2024-04-24T23:49:21.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f3cd6ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ce015a4e0 0x7f3ce013edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48422/0 (socket says 172.21.15.16:48422) 2024-04-24T23:49:21.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f3cd6ffd640 1 -- 172.21.15.16:0/1460538405 learned_addr learned my addr 172.21.15.16:0/1460538405 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f4d417fa640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:6808/542868407,v1:172.21.15.156:6809/542868407] conn(0x7f4d30075680 0x7f4d30077ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f4d417fa640 1 -- 172.21.15.16:0/2448188098 --> [v2:172.21.15.156:6808/542868407,v1:172.21.15.156:6809/542868407] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4d30078190 con 0x7f4d30075680 2024-04-24T23:49:21.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f4d417fa640 1 -- 172.21.15.16:0/2448188098 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4d3808ab60 con 0x7f4d4415dff0 2024-04-24T23:49:21.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f3cdf7fe640 1 --2- 172.21.15.16:0/1460538405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce006b620 0x7f3ce01660d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f8e6a59c640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e640df220 0x7f8e640d18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f8e6a59c640 1 -- 172.21.15.16:0/1129702946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e64053c70 msgr2=0x7f8e640498a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f8e6a59c640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e64053c70 0x7f8e640498a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.155+0000 7f4d48f13640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4d3006bf00 0x7f4d3006e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f8e6a59c640 1 -- 172.21.15.16:0/1129702946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e640d1df0 con 0x7f8e64033ef0 2024-04-24T23:49:21.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f8e69d9b640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e64053c70 0x7f8e640498a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:21.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f8e6ad9d640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e640df220 0x7f8e640d18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:21.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f3cd6ffd640 1 -- 172.21.15.16:0/1460538405 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3ce015afd0 msgr2=0x7f3ce013f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f3cd6ffd640 1 --2- 172.21.15.16:0/1460538405 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3ce015afd0 0x7f3ce013f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f3cd6ffd640 1 -- 172.21.15.16:0/1460538405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce006b620 msgr2=0x7f3ce01660d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f3cd6ffd640 1 --2- 172.21.15.16:0/1460538405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce006b620 0x7f3ce01660d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f3cd6ffd640 1 -- 172.21.15.16:0/1460538405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ce013afc0 con 0x7f3ce015a4e0 2024-04-24T23:49:21.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7fad8659c640 1 --2- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fad7406bf00 0x7fad7406e3c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fad78034000 tx=0x7fad7802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f3cdf7fe640 1 --2- 172.21.15.16:0/1460538405 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ce006b620 0x7f3ce01660d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:21.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f4d43fff640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:6808/542868407,v1:172.21.15.156:6809/542868407] conn(0x7f4d30075680 0x7f4d30077ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f8e6a59c640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e64033ef0 0x7f8e64175570 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c06a510 tx=0x7f8e6c0737c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7fad8759e640 1 --2- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:6824/2447858915,v1:172.21.15.156:6825/2447858915] conn(0x7fad74075670 0x7fad74077ab0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f4d48f13640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4d3006bf00 0x7f4d3006e3c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4d440c97b0 tx=0x7f4d34008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f7b63fff640 1 -- 172.21.15.16:0/1235783943 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7b78094bf0 con 0x7f7b680051c0 2024-04-24T23:49:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f8e5affd640 1 -- 172.21.15.16:0/1129702946 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c073a90 con 0x7f8e64033ef0 2024-04-24T23:49:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f3cd6ffd640 1 --2- 172.21.15.16:0/1460538405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ce015a4e0 0x7f3ce013edd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3cc800c460 tx=0x7f3cc800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f8e5affd640 1 -- 172.21.15.16:0/1129702946 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e6c073c30 con 0x7f8e64033ef0 2024-04-24T23:49:21.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.156+0000 7f7b63fff640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7b5c06bf00 0x7f7b5c06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.157+0000 7f7b63fff640 1 -- 172.21.15.16:0/1235783943 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f7b7810ebf0 con 0x7f7b680051c0 2024-04-24T23:49:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.157+0000 7f3cdcff9640 1 -- 172.21.15.16:0/1460538405 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8005480 con 0x7f3ce015a4e0 2024-04-24T23:49:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.157+0000 7f3ce4a58640 1 -- 172.21.15.16:0/1460538405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ce013b2a0 con 0x7f3ce015a4e0 2024-04-24T23:49:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.157+0000 7f3cdcff9640 1 -- 172.21.15.16:0/1460538405 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cc8005480 con 0x7f3ce015a4e0 2024-04-24T23:49:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.157+0000 7f7b63fff640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:6800/1918945722,v1:172.21.15.156:6801/1918945722] conn(0x7f7b5c075680 0x7f7b5c077ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.157+0000 7f7b63fff640 1 -- 172.21.15.16:0/1235783943 --> [v2:172.21.15.156:6800/1918945722,v1:172.21.15.156:6801/1918945722] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b5c078190 con 0x7f7b5c075680 2024-04-24T23:49:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.157+0000 7f7b63fff640 1 -- 172.21.15.16:0/1235783943 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7b7810efe0 con 0x7f7b680051c0 2024-04-24T23:49:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.157+0000 7f8e5affd640 1 -- 172.21.15.16:0/1129702946 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c074020 con 0x7f8e64033ef0 2024-04-24T23:49:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.157+0000 7f8e6b59e640 1 -- 172.21.15.16:0/1129702946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e641780f0 con 0x7f8e64033ef0 2024-04-24T23:49:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.157+0000 7f8e6b59e640 1 -- 172.21.15.16:0/1129702946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e641786b0 con 0x7f8e64033ef0 2024-04-24T23:49:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.157+0000 7f3ce4a58640 1 -- 172.21.15.16:0/1460538405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ce013b860 con 0x7f3ce015a4e0 2024-04-24T23:49:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.158+0000 7f3cdcff9640 1 -- 172.21.15.16:0/1460538405 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc80035f0 con 0x7f3ce015a4e0 2024-04-24T23:49:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.158+0000 7f7b76ffd640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7b5c06bf00 0x7f7b5c06e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.158+0000 7fad6f7fe640 1 -- 172.21.15.16:0/2799282109 <== osd.7 v2:172.21.15.156:6824/2447858915 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fad74078180 con 0x7fad74075670 2024-04-24T23:49:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.158+0000 7f3cd5ffb640 1 -- 172.21.15.16:0/1460538405 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3cb8001030 con 0x7f3ce015a4e0 2024-04-24T23:49:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.159+0000 7f7b77fff640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:6800/1918945722,v1:172.21.15.156:6801/1918945722] conn(0x7f7b5c075680 0x7f7b5c077ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.159+0000 7f8e6b59e640 1 -- 172.21.15.16:0/1129702946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8e64178d70 con 0x7f8e64033ef0 2024-04-24T23:49:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.159+0000 7f7b77fff640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:6800/1918945722,v1:172.21.15.156:6801/1918945722] conn(0x7f7b5c075680 0x7f7b5c077ac0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.159+0000 7f7b76ffd640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7b5c06bf00 0x7f7b5c06e3c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7b700e6410 tx=0x7f7b6c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.159+0000 7f3cdcff9640 1 -- 172.21.15.16:0/1460538405 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3cc8003790 con 0x7f3ce015a4e0 2024-04-24T23:49:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.159+0000 7f4d43fff640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:6808/542868407,v1:172.21.15.156:6809/542868407] conn(0x7f4d30075680 0x7f4d30077ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.160+0000 7f4d417fa640 1 -- 172.21.15.16:0/2448188098 <== osd.4 v2:172.21.15.156:6808/542868407 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4d30078190 con 0x7f4d30075680 2024-04-24T23:49:21.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.160+0000 7f3cdcff9640 1 --2- 172.21.15.16:0/1460538405 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3cc006c030 0x7f3cc006e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.160+0000 7f3cdcff9640 1 -- 172.21.15.16:0/1460538405 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3cc808aa20 con 0x7f3ce015a4e0 2024-04-24T23:49:21.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.160+0000 7f3cdcff9640 1 --2- 172.21.15.16:0/1460538405 >> [v2:172.21.15.16:6816/3747244008,v1:172.21.15.16:6817/3747244008] conn(0x7f3cc0075730 0x7f3cc0077b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.160+0000 7f3cdcff9640 1 -- 172.21.15.16:0/1460538405 --> [v2:172.21.15.16:6816/3747244008,v1:172.21.15.16:6817/3747244008] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3cc0078240 con 0x7f3cc0075730 2024-04-24T23:49:21.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.160+0000 7f3cdcff9640 1 -- 172.21.15.16:0/1460538405 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3cc808f2a0 con 0x7f3ce015a4e0 2024-04-24T23:49:21.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.160+0000 7f8e5affd640 1 -- 172.21.15.16:0/1129702946 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8e6c0745b0 con 0x7f8e64033ef0 2024-04-24T23:49:21.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.160+0000 7f3cdf7fe640 1 --2- 172.21.15.16:0/1460538405 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3cc006c030 0x7f3cc006e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.160+0000 7f7b63fff640 1 -- 172.21.15.16:0/1235783943 <== osd.5 v2:172.21.15.156:6800/1918945722 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b5c078190 con 0x7f7b5c075680 2024-04-24T23:49:21.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.160+0000 7f3cdffff640 1 --2- 172.21.15.16:0/1460538405 >> [v2:172.21.15.16:6816/3747244008,v1:172.21.15.16:6817/3747244008] conn(0x7f3cc0075730 0x7f3cc0077b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.161+0000 7f8e5affd640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e4c06be30 0x7f8e4c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.161+0000 7f3cdffff640 1 --2- 172.21.15.16:0/1460538405 >> [v2:172.21.15.16:6816/3747244008,v1:172.21.15.16:6817/3747244008] conn(0x7f3cc0075730 0x7f3cc0077b70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.161+0000 7f8e5affd640 1 -- 172.21.15.16:0/1129702946 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8e6c1079d0 con 0x7f8e64033ef0 2024-04-24T23:49:21.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.161+0000 7f8e5affd640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.16:6824/2411432146,v1:172.21.15.16:6825/2411432146] conn(0x7f8e4c0755b0 0x7f8e4c0779f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.161+0000 7f8e5affd640 1 -- 172.21.15.16:0/1129702946 --> [v2:172.21.15.16:6824/2411432146,v1:172.21.15.16:6825/2411432146] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e6c073c30 con 0x7f8e4c0755b0 2024-04-24T23:49:21.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.161+0000 7f8e5affd640 1 -- 172.21.15.16:0/1129702946 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8e6c107dc0 con 0x7f8e64033ef0 2024-04-24T23:49:21.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.162+0000 7f8e6ad9d640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.16:6824/2411432146,v1:172.21.15.16:6825/2411432146] conn(0x7f8e4c0755b0 0x7f8e4c0779f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.162+0000 7f3cdcff9640 1 -- 172.21.15.16:0/1460538405 <== osd.3 v2:172.21.15.16:6816/3747244008 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3cc0078240 con 0x7f3cc0075730 2024-04-24T23:49:21.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.163+0000 7f8e6ad9d640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.16:6824/2411432146,v1:172.21.15.16:6825/2411432146] conn(0x7f8e4c0755b0 0x7f8e4c0779f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.164+0000 7f8e5affd640 1 -- 172.21.15.16:0/1129702946 <== osd.0 v2:172.21.15.16:6824/2411432146 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8e6c073c30 con 0x7f8e4c0755b0 2024-04-24T23:49:21.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.166+0000 7f3cdf7fe640 1 --2- 172.21.15.16:0/1460538405 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3cc006c030 0x7f3cc006e4f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3cd00074c0 tx=0x7f3cd000b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.168+0000 7f8e69d9b640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e4c06be30 0x7f8e4c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.184+0000 7f8e69d9b640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e4c06be30 0x7f8e4c06e2f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8e64049810 tx=0x7f8e60008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.185+0000 7f05a67fc640 1 -- 172.21.15.16:0/2701474111 --> [v2:172.21.15.156:6816/667041555,v1:172.21.15.156:6817/667041555] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f05a0002dd0 con 0x7f05ac0756d0 2024-04-24T23:49:21.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.186+0000 7f05bcff9640 1 -- 172.21.15.16:0/2701474111 <== osd.6 v2:172.21.15.156:6816/667041555 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f05a0002dd0 con 0x7f05ac0756d0 2024-04-24T23:49:21.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.186+0000 7f05c51f6640 1 -- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:6816/667041555,v1:172.21.15.156:6817/667041555] conn(0x7f05ac0756d0 msgr2=0x7f05ac077b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.186+0000 7f05c51f6640 1 --2- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:6816/667041555,v1:172.21.15.156:6817/667041555] conn(0x7f05ac0756d0 0x7f05ac077b10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.186+0000 7f05c51f6640 1 -- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f05ac06bf60 msgr2=0x7f05ac06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.186+0000 7f05c51f6640 1 --2- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f05ac06bf60 0x7f05ac06e420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f05b0034000 tx=0x7f05b002e000 comp rx=0 tx=0).stop 2024-04-24T23:49:21.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.187+0000 7f05c51f6640 1 -- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05c015ddf0 msgr2=0x7f05c0147260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.187+0000 7f05c51f6640 1 --2- 172.21.15.16:0/2701474111 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05c015ddf0 0x7f05c0147260 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f05b800ba80 tx=0x7f05b800bf40 comp rx=0 tx=0).stop 2024-04-24T23:49:21.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.187+0000 7f05c49f5640 1 -- 172.21.15.16:0/2701474111 reap_dead start 2024-04-24T23:49:21.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.187+0000 7f05c51f6640 1 -- 172.21.15.16:0/2701474111 shutdown_connections 2024-04-24T23:49:21.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.187+0000 7f05c51f6640 1 -- 172.21.15.16:0/2701474111 >> 172.21.15.16:0/2701474111 conn(0x7f05c00d4060 msgr2=0x7f05c014b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.187+0000 7f1fce058640 1 -- 172.21.15.16:0/1568975994 --> [v2:172.21.15.16:6802/4175732037,v1:172.21.15.16:6804/4175732037] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1fc0006d90 con 0x7f1fac075630 2024-04-24T23:49:21.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.187+0000 7f05c51f6640 1 -- 172.21.15.16:0/2701474111 shutdown_connections 2024-04-24T23:49:21.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.187+0000 7f05c51f6640 1 -- 172.21.15.16:0/2701474111 wait complete. 2024-04-24T23:49:21.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.188+0000 7f1fb3fff640 1 -- 172.21.15.16:0/1568975994 <== osd.2 v2:172.21.15.16:6802/4175732037 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1fc0006d90 con 0x7f1fac075630 2024-04-24T23:49:21.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.188+0000 7f8fea010640 1 -- 172.21.15.16:0/2721598489 --> [v2:172.21.15.16:6800/2543873173,v1:172.21.15.16:6801/2543873173] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8fdc0df1e0 con 0x7f8fc8075680 2024-04-24T23:49:21.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.188+0000 7f1fce058640 1 -- 172.21.15.16:0/1568975994 >> [v2:172.21.15.16:6802/4175732037,v1:172.21.15.16:6804/4175732037] conn(0x7f1fac075630 msgr2=0x7f1fac077a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.188+0000 7f1fce058640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.16:6802/4175732037,v1:172.21.15.16:6804/4175732037] conn(0x7f1fac075630 0x7f1fac077a70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.188+0000 7f1fce058640 1 -- 172.21.15.16:0/1568975994 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1fac06beb0 msgr2=0x7f1fac06e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.188+0000 7f1fce058640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1fac06beb0 0x7f1fac06e370 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f1fb4034000 tx=0x7f1fb402e000 comp rx=0 tx=0).stop 2024-04-24T23:49:21.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.188+0000 7f1fce058640 1 -- 172.21.15.16:0/1568975994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc003e830 msgr2=0x7f1fc00d5f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.188+0000 7f1fce058640 1 --2- 172.21.15.16:0/1568975994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1fc003e830 0x7f1fc00d5f50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00b780 tx=0x7f1fbc00bc40 comp rx=0 tx=0).stop 2024-04-24T23:49:21.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.189+0000 7f8fd9ffb640 1 -- 172.21.15.16:0/2721598489 <== osd.1 v2:172.21.15.16:6800/2543873173 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8fdc0df1e0 con 0x7f8fc8075680 2024-04-24T23:49:21.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.189+0000 7f1fc7fff640 1 -- 172.21.15.16:0/1568975994 reap_dead start 2024-04-24T23:49:21.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.189+0000 7f1fce058640 1 -- 172.21.15.16:0/1568975994 shutdown_connections 2024-04-24T23:49:21.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.189+0000 7f1fce058640 1 -- 172.21.15.16:0/1568975994 >> 172.21.15.16:0/1568975994 conn(0x7f1fc00559d0 msgr2=0x7f1fc00578c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.189+0000 7f8fea010640 1 -- 172.21.15.16:0/2721598489 >> [v2:172.21.15.16:6800/2543873173,v1:172.21.15.16:6801/2543873173] conn(0x7f8fc8075680 msgr2=0x7f8fc8077ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.189+0000 7f8fea010640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.16:6800/2543873173,v1:172.21.15.16:6801/2543873173] conn(0x7f8fc8075680 0x7f8fc8077ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.189+0000 7f1fce058640 1 -- 172.21.15.16:0/1568975994 shutdown_connections 2024-04-24T23:49:21.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.189+0000 7f1fce058640 1 -- 172.21.15.16:0/1568975994 wait complete. 2024-04-24T23:49:21.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.189+0000 7f8fea010640 1 -- 172.21.15.16:0/2721598489 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8fc806bf00 msgr2=0x7f8fc806e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.189+0000 7f8fea010640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8fc806bf00 0x7f8fc806e3c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8fe402d700 tx=0x7f8fe40ab000 comp rx=0 tx=0).stop 2024-04-24T23:49:21.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.189+0000 7f8fea010640 1 -- 172.21.15.16:0/2721598489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fdc0e3160 msgr2=0x7f8fdc180b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.189+0000 7f8fea010640 1 --2- 172.21.15.16:0/2721598489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fdc0e3160 0x7f8fdc180b40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000c440 tx=0x7f8fe000c900 comp rx=0 tx=0).stop 2024-04-24T23:49:21.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.189+0000 7f8fe980f640 1 -- 172.21.15.16:0/2721598489 reap_dead start 2024-04-24T23:49:21.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.190+0000 7f8fea010640 1 -- 172.21.15.16:0/2721598489 shutdown_connections 2024-04-24T23:49:21.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.190+0000 7f8fea010640 1 -- 172.21.15.16:0/2721598489 >> 172.21.15.16:0/2721598489 conn(0x7f8fdc058ee0 msgr2=0x7f8fdc03b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.190+0000 7f8fea010640 1 -- 172.21.15.16:0/2721598489 shutdown_connections 2024-04-24T23:49:21.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.190+0000 7f8fea010640 1 -- 172.21.15.16:0/2721598489 wait complete. 2024-04-24T23:49:21.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.192+0000 7fad8d07d640 1 -- 172.21.15.16:0/2799282109 --> [v2:172.21.15.156:6824/2447858915,v1:172.21.15.156:6825/2447858915] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fad8813f290 con 0x7fad74075670 2024-04-24T23:49:21.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.192+0000 7fad6f7fe640 1 -- 172.21.15.16:0/2799282109 <== osd.7 v2:172.21.15.156:6824/2447858915 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fad8813f290 con 0x7fad74075670 2024-04-24T23:49:21.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.193+0000 7f7b7c8f8640 1 -- 172.21.15.16:0/1235783943 --> [v2:172.21.15.156:6800/1918945722,v1:172.21.15.156:6801/1918945722] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7b700e2d20 con 0x7f7b5c075680 2024-04-24T23:49:21.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.193+0000 7f4d4b177640 1 -- 172.21.15.16:0/2448188098 --> [v2:172.21.15.156:6808/542868407,v1:172.21.15.156:6809/542868407] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4d24002dd0 con 0x7f4d30075680 2024-04-24T23:49:21.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.193+0000 7f7b63fff640 1 -- 172.21.15.16:0/1235783943 <== osd.5 v2:172.21.15.156:6800/1918945722 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7b700e2d20 con 0x7f7b5c075680 2024-04-24T23:49:21.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.193+0000 7fad6d7fa640 1 -- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:6824/2447858915,v1:172.21.15.156:6825/2447858915] conn(0x7fad74075670 msgr2=0x7fad74077ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.193+0000 7fad6d7fa640 1 --2- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:6824/2447858915,v1:172.21.15.156:6825/2447858915] conn(0x7fad74075670 0x7fad74077ab0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7f4d417fa640 1 -- 172.21.15.16:0/2448188098 <== osd.4 v2:172.21.15.156:6808/542868407 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f4d24002dd0 con 0x7f4d30075680 2024-04-24T23:49:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7fad6d7fa640 1 -- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fad7406bf00 msgr2=0x7fad7406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7fad6d7fa640 1 --2- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fad7406bf00 0x7fad7406e3c0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fad78034000 tx=0x7fad7802e000 comp rx=0 tx=0).stop 2024-04-24T23:49:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7fad6d7fa640 1 -- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad880b4ac0 msgr2=0x7fad8813a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7fad6d7fa640 1 --2- 172.21.15.16:0/2799282109 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fad880b4ac0 0x7fad8813a320 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fad8000ce40 tx=0x7fad8000a660 comp rx=0 tx=0).stop 2024-04-24T23:49:21.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7fad8759e640 1 -- 172.21.15.16:0/2799282109 reap_dead start 2024-04-24T23:49:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7f7b61ffb640 1 -- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:6800/1918945722,v1:172.21.15.156:6801/1918945722] conn(0x7f7b5c075680 msgr2=0x7f7b5c077ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7f7b61ffb640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:6800/1918945722,v1:172.21.15.156:6801/1918945722] conn(0x7f7b5c075680 0x7f7b5c077ac0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7fad6d7fa640 1 -- 172.21.15.16:0/2799282109 shutdown_connections 2024-04-24T23:49:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7f7b61ffb640 1 -- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7b5c06bf00 msgr2=0x7f7b5c06e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7f7b61ffb640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7b5c06bf00 0x7f7b5c06e3c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7b700e6410 tx=0x7f7b6c008040 comp rx=0 tx=0).stop 2024-04-24T23:49:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7f7b61ffb640 1 -- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7b680051c0 msgr2=0x7f7b70038840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7f7b61ffb640 1 --2- 172.21.15.16:0/1235783943 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7b680051c0 0x7f7b70038840 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7b78050ce0 tx=0x7f7b78068530 comp rx=0 tx=0).stop 2024-04-24T23:49:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7f7b77fff640 1 -- 172.21.15.16:0/1235783943 reap_dead start 2024-04-24T23:49:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7fad6d7fa640 1 -- 172.21.15.16:0/2799282109 >> 172.21.15.16:0/2799282109 conn(0x7fad880af800 msgr2=0x7fad8813abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7f7b61ffb640 1 -- 172.21.15.16:0/1235783943 shutdown_connections 2024-04-24T23:49:21.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7f4d4b177640 1 -- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:6808/542868407,v1:172.21.15.156:6809/542868407] conn(0x7f4d30075680 msgr2=0x7f4d30077ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7f4d4b177640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:6808/542868407,v1:172.21.15.156:6809/542868407] conn(0x7f4d30075680 0x7f4d30077ac0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.194+0000 7f7b61ffb640 1 -- 172.21.15.16:0/1235783943 >> 172.21.15.16:0/1235783943 conn(0x7f7b70058ee0 msgr2=0x7f7b7003c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.195+0000 7fad6d7fa640 1 -- 172.21.15.16:0/2799282109 shutdown_connections 2024-04-24T23:49:21.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.195+0000 7fad6d7fa640 1 -- 172.21.15.16:0/2799282109 wait complete. 2024-04-24T23:49:21.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.195+0000 7f4d4b177640 1 -- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4d3006bf00 msgr2=0x7f4d3006e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.195+0000 7f4d4b177640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4d3006bf00 0x7f4d3006e3c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4d440c97b0 tx=0x7f4d34008040 comp rx=0 tx=0).stop 2024-04-24T23:49:21.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.195+0000 7f4d4b177640 1 -- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d4415dff0 msgr2=0x7f4d440c77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.195+0000 7f4d4b177640 1 --2- 172.21.15.16:0/2448188098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d4415dff0 0x7f4d440c77a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4d3800c3f0 tx=0x7f4d3800c8b0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.195+0000 7f4d49714640 1 -- 172.21.15.16:0/2448188098 reap_dead start 2024-04-24T23:49:21.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.195+0000 7f4d4b177640 1 -- 172.21.15.16:0/2448188098 shutdown_connections 2024-04-24T23:49:21.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.195+0000 7f4d4b177640 1 -- 172.21.15.16:0/2448188098 >> 172.21.15.16:0/2448188098 conn(0x7f4d440d06f0 msgr2=0x7f4d4406bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.195+0000 7f4d4b177640 1 -- 172.21.15.16:0/2448188098 shutdown_connections 2024-04-24T23:49:21.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.195+0000 7f4d4b177640 1 -- 172.21.15.16:0/2448188098 wait complete. 2024-04-24T23:49:21.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.196+0000 7f7b61ffb640 1 -- 172.21.15.16:0/1235783943 shutdown_connections 2024-04-24T23:49:21.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.196+0000 7f7b61ffb640 1 -- 172.21.15.16:0/1235783943 wait complete. 2024-04-24T23:49:21.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.196+0000 7f3cd5ffb640 1 -- 172.21.15.16:0/1460538405 --> [v2:172.21.15.16:6816/3747244008,v1:172.21.15.16:6817/3747244008] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3cb8002dd0 con 0x7f3cc0075730 2024-04-24T23:49:21.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.197+0000 7f8e6b59e640 1 -- 172.21.15.16:0/1129702946 --> [v2:172.21.15.16:6824/2411432146,v1:172.21.15.16:6825/2411432146] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8e64005390 con 0x7f8e4c0755b0 2024-04-24T23:49:21.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.197+0000 7f3cdcff9640 1 -- 172.21.15.16:0/1460538405 <== osd.3 v2:172.21.15.16:6816/3747244008 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3cb8002dd0 con 0x7f3cc0075730 2024-04-24T23:49:21.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.198+0000 7f8e5affd640 1 -- 172.21.15.16:0/1129702946 <== osd.0 v2:172.21.15.16:6824/2411432146 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8e64005390 con 0x7f8e4c0755b0 2024-04-24T23:49:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.198+0000 7f3cd5ffb640 1 -- 172.21.15.16:0/1460538405 >> [v2:172.21.15.16:6816/3747244008,v1:172.21.15.16:6817/3747244008] conn(0x7f3cc0075730 msgr2=0x7f3cc0077b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.198+0000 7f3cd5ffb640 1 --2- 172.21.15.16:0/1460538405 >> [v2:172.21.15.16:6816/3747244008,v1:172.21.15.16:6817/3747244008] conn(0x7f3cc0075730 0x7f3cc0077b70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.199+0000 7f8e58ff9640 1 -- 172.21.15.16:0/1129702946 >> [v2:172.21.15.16:6824/2411432146,v1:172.21.15.16:6825/2411432146] conn(0x7f8e4c0755b0 msgr2=0x7f8e4c0779f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.199+0000 7f8e58ff9640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.16:6824/2411432146,v1:172.21.15.16:6825/2411432146] conn(0x7f8e4c0755b0 0x7f8e4c0779f0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.198+0000 7f3cd5ffb640 1 -- 172.21.15.16:0/1460538405 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3cc006c030 msgr2=0x7f3cc006e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.198+0000 7f3cd5ffb640 1 --2- 172.21.15.16:0/1460538405 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3cc006c030 0x7f3cc006e4f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3cd00074c0 tx=0x7f3cd000b040 comp rx=0 tx=0).stop 2024-04-24T23:49:21.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.199+0000 7f3cd5ffb640 1 -- 172.21.15.16:0/1460538405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ce015a4e0 msgr2=0x7f3ce013edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.199+0000 7f3cd5ffb640 1 --2- 172.21.15.16:0/1460538405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ce015a4e0 0x7f3ce013edd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3cc800c460 tx=0x7f3cc800c920 comp rx=0 tx=0).stop 2024-04-24T23:49:21.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.199+0000 7f3cdffff640 1 -- 172.21.15.16:0/1460538405 reap_dead start 2024-04-24T23:49:21.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.199+0000 7f3cd5ffb640 1 -- 172.21.15.16:0/1460538405 shutdown_connections 2024-04-24T23:49:21.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.199+0000 7f3cd5ffb640 1 -- 172.21.15.16:0/1460538405 >> 172.21.15.16:0/1460538405 conn(0x7f3ce00af700 msgr2=0x7f3ce0071b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.201+0000 7f3cd5ffb640 1 -- 172.21.15.16:0/1460538405 shutdown_connections 2024-04-24T23:49:21.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.201+0000 7f3cd5ffb640 1 -- 172.21.15.16:0/1460538405 wait complete. 2024-04-24T23:49:21.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.201+0000 7f8e58ff9640 1 -- 172.21.15.16:0/1129702946 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e4c06be30 msgr2=0x7f8e4c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.201+0000 7f8e58ff9640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e4c06be30 0x7f8e4c06e2f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8e64049810 tx=0x7f8e60008040 comp rx=0 tx=0).stop 2024-04-24T23:49:21.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.201+0000 7f8e58ff9640 1 -- 172.21.15.16:0/1129702946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e64033ef0 msgr2=0x7f8e64175570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.202+0000 7f8e58ff9640 1 --2- 172.21.15.16:0/1129702946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e64033ef0 0x7f8e64175570 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c06a510 tx=0x7f8e6c0737c0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.202+0000 7f8e6ad9d640 1 -- 172.21.15.16:0/1129702946 reap_dead start 2024-04-24T23:49:21.266 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2024-04-24T23:49:21.266 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-24T23:49:21.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.208+0000 7f8e58ff9640 1 -- 172.21.15.16:0/1129702946 shutdown_connections 2024-04-24T23:49:21.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.208+0000 7f8e58ff9640 1 -- 172.21.15.16:0/1129702946 >> 172.21.15.16:0/1129702946 conn(0x7f8e64009fb0 msgr2=0x7f8e64036590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.208+0000 7f8e58ff9640 1 -- 172.21.15.16:0/1129702946 shutdown_connections 2024-04-24T23:49:21.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.208+0000 7f8e58ff9640 1 -- 172.21.15.16:0/1129702946 wait complete. 2024-04-24T23:49:21.268 INFO:teuthology.orchestra.run.smithi016.stdout:55834574859 2024-04-24T23:49:21.268 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T23:49:21.269 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2024-04-24T23:49:21.269 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T23:49:21.270 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2024-04-24T23:49:21.270 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T23:49:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2024-04-24T23:49:21.271 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T23:49:21.272 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2024-04-24T23:49:21.272 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-24T23:49:21.273 INFO:teuthology.orchestra.run.smithi016.stdout:55834574859 2024-04-24T23:49:21.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T23:49:21.274 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2024-04-24T23:49:21.274 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T23:49:21.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.487+0000 7f8c3564d640 1 Processor -- start 2024-04-24T23:49:21.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.488+0000 7f8c3564d640 1 -- start start 2024-04-24T23:49:21.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.489+0000 7f8c3564d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c30148580 0x7f8c300b6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.489+0000 7f8c3564d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8c30166420 0x7f8c30149b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.489+0000 7f8c3564d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c30148960 con 0x7f8c300b70f0 2024-04-24T23:49:21.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.489+0000 7f8c3564d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8c300b7550 con 0x7f8c30148580 2024-04-24T23:49:21.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.489+0000 7f8c3564d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8c301496a0 con 0x7f8c30166420 2024-04-24T23:49:21.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.489+0000 7f8c2effd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c30148580 0x7f8c300b6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.489+0000 7f8c2effd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c30148580 0x7f8c300b6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48426/0 (socket says 172.21.15.16:48426) 2024-04-24T23:49:21.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.489+0000 7f8c2effd640 1 -- 172.21.15.16:0/3440993140 learned_addr learned my addr 172.21.15.16:0/3440993140 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.489+0000 7f18c355c640 1 Processor -- start 2024-04-24T23:49:21.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.489+0000 7f18c355c640 1 -- start start 2024-04-24T23:49:21.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.490+0000 7f8c2f7fe640 1 --2- 172.21.15.16:0/3440993140 >> v2:172.21.15.156:3300/0 conn(0x7f8c30166420 0x7f8c30149b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.490+0000 7f8c2f7fe640 1 -- 172.21.15.16:0/3440993140 >> v2:172.21.15.16:3300/0 conn(0x7f8c30148580 msgr2=0x7f8c300b6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.490+0000 7f8c2f7fe640 1 --2- 172.21.15.16:0/3440993140 >> v2:172.21.15.16:3300/0 conn(0x7f8c30148580 0x7f8c300b6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.490+0000 7f8c2f7fe640 1 -- 172.21.15.16:0/3440993140 >> v1:172.21.15.16:6789/0 conn(0x7f8c300b70f0 legacy=0x7f8c30165d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.490+0000 7f8c2f7fe640 1 -- 172.21.15.16:0/3440993140 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c30166a00 con 0x7f8c30166420 2024-04-24T23:49:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.490+0000 7f8c2effd640 1 --2- 172.21.15.16:0/3440993140 >> v2:172.21.15.16:3300/0 conn(0x7f8c30148580 0x7f8c300b6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:21.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.491+0000 7f8c2f7fe640 1 --2- 172.21.15.16:0/3440993140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c30166420 0x7f8c30149b30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8c30070ab0 tx=0x7f8c2002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f019544bdd790a63 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.491+0000 7f8c2dffb640 1 -- 172.21.15.16:0/3440993140 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c20037070 con 0x7f8c30166420 2024-04-24T23:49:21.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.492+0000 7f8c2dffb640 1 -- 172.21.15.16:0/3440993140 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c20032440 con 0x7f8c30166420 2024-04-24T23:49:21.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.491+0000 7f18c355c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18b4054200 0x7f18b40df800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.491+0000 7f18c355c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18b4038520 0x7f18b40389b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.491+0000 7f18c355c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18b40526a0 con 0x7f18b4050740 2024-04-24T23:49:21.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.491+0000 7f18c355c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f18b4053850 con 0x7f18b4054200 2024-04-24T23:49:21.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.491+0000 7f18c355c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f18b4053c80 con 0x7f18b4038520 2024-04-24T23:49:21.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.491+0000 7f18c0af7640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f18b4050740 0x7f18b4053140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:41302/0 (socket says 172.21.15.16:41302) 2024-04-24T23:49:21.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.491+0000 7f18c0af7640 1 -- 172.21.15.16:0/4207758716 learned_addr learned my addr 172.21.15.16:0/4207758716 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:21.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.493+0000 7f8c3564d640 1 -- 172.21.15.16:0/3440993140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c30166420 msgr2=0x7f8c30149b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.493+0000 7f8c3564d640 1 --2- 172.21.15.16:0/3440993140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c30166420 0x7f8c30149b30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8c30070ab0 tx=0x7f8c2002a1c0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.491+0000 7f18c12f8640 1 --2- 172.21.15.16:0/4207758716 >> v2:172.21.15.16:3300/0 conn(0x7f18b4054200 0x7f18b40df800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.491+0000 7f18c1af9640 1 --2- 172.21.15.16:0/4207758716 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18b4038520 0x7f18b40389b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.493+0000 7f8c3564d640 1 -- 172.21.15.16:0/3440993140 shutdown_connections 2024-04-24T23:49:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.492+0000 7f18c1af9640 1 -- 172.21.15.16:0/4207758716 >> v2:172.21.15.16:3300/0 conn(0x7f18b4054200 msgr2=0x7f18b40df800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.492+0000 7f18c1af9640 1 --2- 172.21.15.16:0/4207758716 >> v2:172.21.15.16:3300/0 conn(0x7f18b4054200 0x7f18b40df800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.492+0000 7f18c1af9640 1 -- 172.21.15.16:0/4207758716 >> v1:172.21.15.156:6789/0 conn(0x7f18b4050740 legacy=0x7f18b4053140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.492+0000 7f18c1af9640 1 -- 172.21.15.16:0/4207758716 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b4052940 con 0x7f18b4038520 2024-04-24T23:49:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.492+0000 7f18b3fff640 1 -- 172.21.15.16:0/4207758716 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3174219508 0 0) 0x7f18b40526a0 con 0x7f18b4050740 2024-04-24T23:49:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.493+0000 7f8c3564d640 1 --2- 172.21.15.16:0/3440993140 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c30166420 0x7f8c30149b30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.493+0000 7f8c3564d640 1 --2- 172.21.15.16:0/3440993140 >> v2:172.21.15.16:3300/0 conn(0x7f8c30148580 0x7f8c300b6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.493+0000 7f8c3564d640 1 -- 172.21.15.16:0/3440993140 >> 172.21.15.16:0/3440993140 conn(0x7f8c3006b500 msgr2=0x7f8c30072a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.493+0000 7f18c1af9640 1 --2- 172.21.15.16:0/4207758716 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18b4038520 0x7f18b40389b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f18bc000b60 tx=0x7f18bc0b6ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec87d05adee49010 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.493+0000 7f8c3564d640 1 -- 172.21.15.16:0/3440993140 shutdown_connections 2024-04-24T23:49:21.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.493+0000 7f18b3fff640 1 -- 172.21.15.16:0/4207758716 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc0ba930 con 0x7f18b4038520 2024-04-24T23:49:21.504 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.493+0000 7f18b3fff640 1 -- 172.21.15.16:0/4207758716 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18bc0bab10 con 0x7f18b4038520 2024-04-24T23:49:21.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.493+0000 7f8c3564d640 1 -- 172.21.15.16:0/3440993140 wait complete. 2024-04-24T23:49:21.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.494+0000 7f18c355c640 1 -- 172.21.15.16:0/4207758716 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18b4038520 msgr2=0x7f18b40389b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.494+0000 7f18c355c640 1 --2- 172.21.15.16:0/4207758716 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18b4038520 0x7f18b40389b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f18bc000b60 tx=0x7f18bc0b6ab0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.494+0000 7fcb8759e640 1 Processor -- start 2024-04-24T23:49:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.495+0000 7f18c355c640 1 -- 172.21.15.16:0/4207758716 shutdown_connections 2024-04-24T23:49:21.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.495+0000 7f18c355c640 1 --2- 172.21.15.16:0/4207758716 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18b4038520 0x7f18b40389b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.495+0000 7f18c355c640 1 --2- 172.21.15.16:0/4207758716 >> v2:172.21.15.16:3300/0 conn(0x7f18b4054200 0x7f18b40df800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.495+0000 7f18c355c640 1 -- 172.21.15.16:0/4207758716 >> 172.21.15.16:0/4207758716 conn(0x7f18b4009fb0 msgr2=0x7f18b400a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.495+0000 7f18c355c640 1 -- 172.21.15.16:0/4207758716 shutdown_connections 2024-04-24T23:49:21.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.495+0000 7f18c355c640 1 -- 172.21.15.16:0/4207758716 wait complete. 2024-04-24T23:49:21.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.495+0000 7f6cd389a640 1 Processor -- start 2024-04-24T23:49:21.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.496+0000 7f8c3564d640 1 Processor -- start 2024-04-24T23:49:21.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.496+0000 7f8c3564d640 1 -- start start 2024-04-24T23:49:21.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.496+0000 7f0af4c1b640 1 Processor -- start 2024-04-24T23:49:21.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.495+0000 7f6cd389a640 1 -- start start 2024-04-24T23:49:21.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.496+0000 7fc43cfc5640 1 Processor -- start 2024-04-24T23:49:21.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.496+0000 7fc43cfc5640 1 -- start start 2024-04-24T23:49:21.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.495+0000 7fcb8759e640 1 -- start start 2024-04-24T23:49:21.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.496+0000 7fcb8759e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb88142a60 0x7fcb8806b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.496+0000 7fcb8759e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fcb8806cdc0 0x7fcb880ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.496+0000 7fcb8759e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb8806b770 con 0x7fcb880ce580 2024-04-24T23:49:21.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.496+0000 7fcb8759e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fcb88142e40 con 0x7fcb88142a60 2024-04-24T23:49:21.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.496+0000 7fcb8759e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fcb880cdbb0 con 0x7fcb8806cdc0 2024-04-24T23:49:21.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7fcb86d9d640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fcb880ce580 0x7fcb880b2a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:41310/0 (socket says 172.21.15.16:41310) 2024-04-24T23:49:21.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7fcb85d9b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fcb8806cdc0 0x7fcb880ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7fcb86d9d640 1 -- 172.21.15.16:0/3238130960 learned_addr learned my addr 172.21.15.16:0/3238130960 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:21.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7fcb8659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb88142a60 0x7fcb8806b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.496+0000 7f0af4c1b640 1 -- start start 2024-04-24T23:49:21.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7f6cd389a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ccc072dc0 0x7f6ccc14b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7fcb85d9b640 1 -- 172.21.15.16:0/3238130960 >> v2:172.21.15.16:3300/0 conn(0x7fcb88142a60 msgr2=0x7fcb8806b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7fcb85d9b640 1 --2- 172.21.15.16:0/3238130960 >> v2:172.21.15.16:3300/0 conn(0x7fcb88142a60 0x7fcb8806b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7f18c355c640 1 Processor -- start 2024-04-24T23:49:21.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7f8c3564d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c300b70f0 0x7f8c301f3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7f6cd0e35640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6ccc166e70 0x7f6ccc162c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50638/0 (socket says 172.21.15.16:50638) 2024-04-24T23:49:21.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7f8c3564d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c30148580 0x7f8c30204c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7fcb85d9b640 1 -- 172.21.15.16:0/3238130960 >> v1:172.21.15.156:6789/0 conn(0x7fcb880ce580 legacy=0x7fcb880b2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7fcb85d9b640 1 -- 172.21.15.16:0/3238130960 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb880cd280 con 0x7fcb8806cdc0 2024-04-24T23:49:21.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7fcb8559a640 1 -- 172.21.15.16:0/3238130960 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4270069979 0 0) 0x7fcb8806b770 con 0x7fcb880ce580 2024-04-24T23:49:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7f0af4c1b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0ae80df420 0x7f0ae8053260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7f0af4c1b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ae80392d0 0x7f0ae80537a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7f0af4c1b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae8006af0 con 0x7f0ae804fa90 2024-04-24T23:49:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7f0af4c1b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae80068a0 con 0x7f0ae80392d0 2024-04-24T23:49:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7f0af4c1b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae8006dd0 con 0x7f0ae80df420 2024-04-24T23:49:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7f0aeffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ae80392d0 0x7f0ae80537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7f0aeffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0ae80392d0 0x7f0ae80537a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48456/0 (socket says 172.21.15.16:48456) 2024-04-24T23:49:21.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7f6cd0e35640 1 -- 172.21.15.16:0/3692464151 learned_addr learned my addr 172.21.15.16:0/3692464151 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:21.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7f0aeffff640 1 -- 172.21.15.16:0/1311439588 learned_addr learned my addr 172.21.15.16:0/1311439588 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7fc43cfc5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4300e1310 0x7fc43003a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7f0aeeffd640 1 --2- 172.21.15.16:0/1311439588 >> v2:172.21.15.156:3300/0 conn(0x7f0ae80df420 0x7f0ae8053260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7f6cd389a640 1 -- 172.21.15.16:0/3692464151 --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ccc1639a0 con 0x7f6ccc166e70 2024-04-24T23:49:21.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7fc43cfc5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc43000ab40 0x7fc430009d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7f0aeffff640 1 -- 172.21.15.16:0/1311439588 >> v2:172.21.15.156:3300/0 conn(0x7f0ae80df420 msgr2=0x7f0ae8053260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7f0aeffff640 1 --2- 172.21.15.16:0/1311439588 >> v2:172.21.15.156:3300/0 conn(0x7f0ae80df420 0x7f0ae8053260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7f0aeffff640 1 -- 172.21.15.16:0/1311439588 >> v1:172.21.15.16:6789/0 conn(0x7f0ae804fa90 legacy=0x7f0ae804fe70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:49:21.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7f9319185640 1 Processor -- start 2024-04-24T23:49:21.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7f9319185640 1 -- start start 2024-04-24T23:49:21.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7fc4377fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4300e1310 0x7fc43003a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7fc4377fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4300e1310 0x7fc43003a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:41494/0 (socket says 172.21.15.16:41494) 2024-04-24T23:49:21.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7fc436ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc43000ab40 0x7fc430009d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7fc436ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc43000ab40 0x7fc430009d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:34896/0 (socket says 172.21.15.16:34896) 2024-04-24T23:49:21.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7fc43cfc5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc43003a6b0 con 0x7fc4300516e0 2024-04-24T23:49:21.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7fcb85d9b640 1 --2- 172.21.15.16:0/3238130960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb8806cdc0 0x7fcb880ce040 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcb8815d8a0 tx=0x7fcb8002c770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f95c368c44a9fff server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7fc43cfc5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fc430055920 con 0x7fc43000ab40 2024-04-24T23:49:21.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.497+0000 7f18c355c640 1 -- start start 2024-04-24T23:49:21.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7fc43cfc5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fc430054d10 con 0x7fc4300e1310 2024-04-24T23:49:21.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7fc4377fe640 1 -- 172.21.15.16:0/2799398445 learned_addr learned my addr 172.21.15.16:0/2799398445 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f8c3564d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8c30166420 0x7f8c30141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f18c355c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18b4038520 0x7f18b4174740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f8c3564d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8c3014a070 con 0x7f8c300b70f0 2024-04-24T23:49:21.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f8c3564d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8c30149330 con 0x7f8c30148580 2024-04-24T23:49:21.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f8c3564d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8c3014a1f0 con 0x7f8c30166420 2024-04-24T23:49:21.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f18c355c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18b4050740 0x7f18b40e1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f8c2f7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8c30166420 0x7f8c30141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f8c2f7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8c30166420 0x7f8c30141740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:41502/0 (socket says 172.21.15.16:41502) 2024-04-24T23:49:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f6cd389a640 1 -- 172.21.15.16:0/3692464151 --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ccc166390 con 0x7f6ccc0731a0 2024-04-24T23:49:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f6cd389a640 1 -- 172.21.15.16:0/3692464151 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6ccc166140 con 0x7f6ccc072dc0 2024-04-24T23:49:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7fc4377fe640 1 -- 172.21.15.16:0/2799398445 >> v2:172.21.15.156:3300/0 conn(0x7fc43000ab40 msgr2=0x7fc430009d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7fc4377fe640 1 --2- 172.21.15.16:0/2799398445 >> v2:172.21.15.156:3300/0 conn(0x7fc43000ab40 0x7fc430009d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7fc4377fe640 1 -- 172.21.15.16:0/2799398445 >> v1:172.21.15.156:6789/0 conn(0x7fc4300516e0 legacy=0x7fc430055db0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:49:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f9319185640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9314143910 0x7f93140720a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f6cd1636640 1 --2- 172.21.15.16:0/3692464151 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ccc072dc0 0x7f6ccc14b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.498+0000 7f0aeffff640 1 -- 172.21.15.16:0/1311439588 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae80070b0 con 0x7f0ae80392d0 2024-04-24T23:49:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7fc4377fe640 1 -- 172.21.15.16:0/2799398445 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc430054ff0 con 0x7fc4300e1310 2024-04-24T23:49:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f8c2effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c300b70f0 0x7f8c301f3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f8c2f7fe640 1 -- 172.21.15.16:0/4051269241 learned_addr learned my addr 172.21.15.16:0/4051269241 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7fcb8559a640 1 -- 172.21.15.16:0/3238130960 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80038070 con 0x7fcb8806cdc0 2024-04-24T23:49:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f9319185640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f93140725e0 0x7f931406e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f8c2e7fc640 1 --2- 172.21.15.16:0/4051269241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c30148580 0x7f8c30204c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f6cc3fff640 1 -- 172.21.15.16:0/3692464151 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1794663796 0 0) 0x7f6ccc1639a0 con 0x7f6ccc166e70 2024-04-24T23:49:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f6cc3fff640 1 -- 172.21.15.16:0/3692464151 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6cb8003520 con 0x7f6ccc166e70 2024-04-24T23:49:21.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f9319185640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f931406e8c0 0x7f931415da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7fcb8559a640 1 -- 172.21.15.16:0/3238130960 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb80031710 con 0x7fcb8806cdc0 2024-04-24T23:49:21.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f9319185640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9314072a50 con 0x7f93140725e0 2024-04-24T23:49:21.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f9319185640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f931406d190 con 0x7f9314143910 2024-04-24T23:49:21.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7fcb8559a640 1 -- 172.21.15.16:0/3238130960 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80031ac0 con 0x7fcb8806cdc0 2024-04-24T23:49:21.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f9319185640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f931406d470 con 0x7f931406e8c0 2024-04-24T23:49:21.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f18c12f8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18b4038520 0x7f18b4174740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f18c355c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18b4054200 0x7f18b40ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f18c355c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f18b4053f60 con 0x7f18b4038520 2024-04-24T23:49:21.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f18c0af7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18b4050740 0x7f18b40e1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f0aeffff640 1 --2- 172.21.15.16:0/1311439588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae80392d0 0x7f0ae80537a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0af002d700 tx=0x7f0af00a5700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3521842d828117f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f9313fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9314143910 0x7f93140720a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f9313fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9314143910 0x7f93140720a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:34912/0 (socket says 172.21.15.16:34912) 2024-04-24T23:49:21.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f9313fff640 1 -- 172.21.15.16:0/2481109477 learned_addr learned my addr 172.21.15.16:0/2481109477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f93137fe640 1 --2- 172.21.15.16:0/2481109477 >> v2:172.21.15.16:3300/0 conn(0x7f93140725e0 0x7f931406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f0aee7fc640 1 -- 172.21.15.16:0/1311439588 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0af00aa7d0 con 0x7f0ae80392d0 2024-04-24T23:49:21.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f18c12f8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18b4038520 0x7f18b4174740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48474/0 (socket says 172.21.15.16:48474) 2024-04-24T23:49:21.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f18c12f8640 1 -- 172.21.15.16:0/3064651321 learned_addr learned my addr 172.21.15.16:0/3064651321 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f18c355c640 1 -- 172.21.15.16:0/3064651321 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f18b4054850 con 0x7f18b4050740 2024-04-24T23:49:21.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f18c355c640 1 -- 172.21.15.16:0/3064651321 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f18b4038ef0 con 0x7f18b4054200 2024-04-24T23:49:21.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f0aee7fc640 1 -- 172.21.15.16:0/1311439588 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0af00aa970 con 0x7f0ae80392d0 2024-04-24T23:49:21.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f8c2effd640 1 -- 172.21.15.16:0/4051269241 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8c30166420 msgr2=0x7f8c30141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f8c2effd640 1 --2- 172.21.15.16:0/4051269241 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8c30166420 0x7f8c30141740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f8c2effd640 1 -- 172.21.15.16:0/4051269241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c30148580 msgr2=0x7f8c30204c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f8c2effd640 1 --2- 172.21.15.16:0/4051269241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c30148580 0x7f8c30204c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.499+0000 7f8c2effd640 1 -- 172.21.15.16:0/4051269241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c30155280 con 0x7f8c300b70f0 2024-04-24T23:49:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f8c2effd640 1 --2- 172.21.15.16:0/4051269241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c300b70f0 0x7f8c301f3d20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800eec0 tx=0x7f8c2800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f8c177fe640 1 -- 172.21.15.16:0/4051269241 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c2800bd90 con 0x7f8c300b70f0 2024-04-24T23:49:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f8c177fe640 1 -- 172.21.15.16:0/4051269241 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c28003990 con 0x7f8c300b70f0 2024-04-24T23:49:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f8c177fe640 1 -- 172.21.15.16:0/4051269241 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c28003d80 con 0x7f8c300b70f0 2024-04-24T23:49:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f18c1af9640 1 --2- 172.21.15.16:0/3064651321 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18b4054200 0x7f18b40ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f18c12f8640 1 -- 172.21.15.16:0/3064651321 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18b4054200 msgr2=0x7f18b40ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f18c12f8640 1 --2- 172.21.15.16:0/3064651321 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18b4054200 0x7f18b40ce6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f18c12f8640 1 -- 172.21.15.16:0/3064651321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18b4050740 msgr2=0x7f18b40e1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f18c12f8640 1 --2- 172.21.15.16:0/3064651321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18b4050740 0x7f18b40e1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f18c12f8640 1 -- 172.21.15.16:0/3064651321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b40e2950 con 0x7f18b4038520 2024-04-24T23:49:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f6cc3fff640 1 -- 172.21.15.16:0/3692464151 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3614276246 0 0) 0x7f6cb8003520 con 0x7f6ccc166e70 2024-04-24T23:49:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f6cc3fff640 1 -- 172.21.15.16:0/3692464151 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ccc072dc0 msgr2=0x7f6ccc14b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f6cc3fff640 1 --2- 172.21.15.16:0/3692464151 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6ccc072dc0 0x7f6ccc14b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f6cc3fff640 1 -- 172.21.15.16:0/3692464151 >> v1:172.21.15.156:6789/0 conn(0x7f6ccc0731a0 legacy=0x7f6ccc0b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f6cc3fff640 1 -- 172.21.15.16:0/3692464151 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ccc166670 con 0x7f6ccc166e70 2024-04-24T23:49:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f6cc3fff640 1 -- 172.21.15.16:0/3692464151 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f6cc40019f0 con 0x7f6ccc166e70 2024-04-24T23:49:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f6cc3fff640 1 -- 172.21.15.16:0/3692464151 >> v1:172.21.15.16:6789/0 conn(0x7f6ccc166e70 legacy=0x7f6ccc162c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f6cc3fff640 1 --2- 172.21.15.16:0/3692464151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cb80039e0 0x7f6cb8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f6cc3fff640 1 --2- 172.21.15.16:0/3692464151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb8004800 0x7f6cb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f6cc3fff640 1 --2- 172.21.15.16:0/3692464151 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb80051c0 0x7f6cb800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f6cc3fff640 1 -- 172.21.15.16:0/3692464151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ccc166140 con 0x7f6cb8004800 2024-04-24T23:49:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f6cc3fff640 1 -- 172.21.15.16:0/3692464151 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb800c090 con 0x7f6cb80039e0 2024-04-24T23:49:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f6cc3fff640 1 -- 172.21.15.16:0/3692464151 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6cb800c210 con 0x7f6cb80051c0 2024-04-24T23:49:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f6cd0e35640 1 --2- 172.21.15.16:0/3692464151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb8004800 0x7f6cb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7fcb8759e640 1 -- 172.21.15.16:0/3238130960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb8806cdc0 msgr2=0x7fcb880ce040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7fcb8759e640 1 --2- 172.21.15.16:0/3238130960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb8806cdc0 0x7fcb880ce040 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcb8815d8a0 tx=0x7fcb8002c770 comp rx=0 tx=0).stop 2024-04-24T23:49:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7fcb8759e640 1 -- 172.21.15.16:0/3238130960 shutdown_connections 2024-04-24T23:49:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7fcb8759e640 1 --2- 172.21.15.16:0/3238130960 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb8806cdc0 0x7fcb880ce040 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcb8815d8a0 tx=0x7fcb8002c770 comp rx=0 tx=0).stop 2024-04-24T23:49:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7fcb8759e640 1 --2- 172.21.15.16:0/3238130960 >> v2:172.21.15.16:3300/0 conn(0x7fcb88142a60 0x7fcb8806b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7fcb8759e640 1 -- 172.21.15.16:0/3238130960 >> 172.21.15.16:0/3238130960 conn(0x7fcb880af700 msgr2=0x7fcb880afb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7fcb8759e640 1 -- 172.21.15.16:0/3238130960 shutdown_connections 2024-04-24T23:49:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7fcb8759e640 1 -- 172.21.15.16:0/3238130960 wait complete. 2024-04-24T23:49:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7fbf3b858640 1 Processor -- start 2024-04-24T23:49:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7fbf3b858640 1 -- start start 2024-04-24T23:49:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f0af4c1b640 1 -- 172.21.15.16:0/1311439588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae80392d0 msgr2=0x7f0ae80537a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f0af4c1b640 1 --2- 172.21.15.16:0/1311439588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae80392d0 0x7f0ae80537a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0af002d700 tx=0x7f0af00a5700 comp rx=0 tx=0).stop 2024-04-24T23:49:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f9313fff640 1 -- 172.21.15.16:0/2481109477 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f931406e8c0 msgr2=0x7f931415da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f9313fff640 1 --2- 172.21.15.16:0/2481109477 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f931406e8c0 0x7f931415da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f9313fff640 1 -- 172.21.15.16:0/2481109477 >> v2:172.21.15.16:3300/0 conn(0x7f93140725e0 msgr2=0x7f931406e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f9313fff640 1 --2- 172.21.15.16:0/2481109477 >> v2:172.21.15.16:3300/0 conn(0x7f93140725e0 0x7f931406e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.500+0000 7f9313fff640 1 -- 172.21.15.16:0/2481109477 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f931406d750 con 0x7f9314143910 2024-04-24T23:49:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f18c0af7640 1 --2- 172.21.15.16:0/3064651321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18b4050740 0x7f18b40e1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f18c12f8640 1 --2- 172.21.15.16:0/3064651321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18b4038520 0x7f18b4174740 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00e9e0 tx=0x7f18ac00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f6cd0e35640 1 -- 172.21.15.16:0/3692464151 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb80051c0 msgr2=0x7f6cb800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f6cd0e35640 1 --2- 172.21.15.16:0/3692464151 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb80051c0 0x7f6cb800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f6cd0e35640 1 -- 172.21.15.16:0/3692464151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cb80039e0 msgr2=0x7f6cb8003e00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:49:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f6cd0e35640 1 --2- 172.21.15.16:0/3692464151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6cb80039e0 0x7f6cb8003e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f6cd0e35640 1 -- 172.21.15.16:0/3692464151 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb8003520 con 0x7f6cb8004800 2024-04-24T23:49:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7fc4377fe640 1 --2- 172.21.15.16:0/2799398445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4300e1310 0x7fc43003a170 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc42400dc90 tx=0x7fc42402e500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a569ad76e4772a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f0af4c1b640 1 -- 172.21.15.16:0/1311439588 shutdown_connections 2024-04-24T23:49:21.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f0af4c1b640 1 --2- 172.21.15.16:0/1311439588 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae80392d0 0x7f0ae80537a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f0af4c1b640 1 --2- 172.21.15.16:0/1311439588 >> v2:172.21.15.156:3300/0 conn(0x7f0ae80df420 0x7f0ae8053260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f0af4c1b640 1 -- 172.21.15.16:0/1311439588 >> 172.21.15.16:0/1311439588 conn(0x7f0ae8009fb0 msgr2=0x7f0ae800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7f0af4c1b640 1 -- 172.21.15.16:0/1311439588 shutdown_connections 2024-04-24T23:49:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7f8c3564d640 1 -- 172.21.15.16:0/4051269241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c301f4370 con 0x7f8c300b70f0 2024-04-24T23:49:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7f8c3564d640 1 -- 172.21.15.16:0/4051269241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c301f4930 con 0x7f8c300b70f0 2024-04-24T23:49:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7fbf3b858640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbf3415e020 0x7fbf340cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7fbf3b858640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbf340d1fa0 0x7fbf340d23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7fbf3b858640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf340b3390 con 0x7fbf340b2bb0 2024-04-24T23:49:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7fbf3b858640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fbf3406bfc0 con 0x7fbf340d1fa0 2024-04-24T23:49:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7fbf3b858640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fbf3406c2a0 con 0x7fbf3415e020 2024-04-24T23:49:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7f0af4c1b640 1 -- 172.21.15.16:0/1311439588 wait complete. 2024-04-24T23:49:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7f18b1ffb640 1 -- 172.21.15.16:0/3064651321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac017070 con 0x7f18b4038520 2024-04-24T23:49:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7fbf39df5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbf340d1fa0 0x7fbf340d23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7fbf39df5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbf340d1fa0 0x7fbf340d23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48500/0 (socket says 172.21.15.16:48500) 2024-04-24T23:49:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7fbf39df5640 1 -- 172.21.15.16:0/889073818 learned_addr learned my addr 172.21.15.16:0/889073818 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7f18b1ffb640 1 -- 172.21.15.16:0/3064651321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18ac00c9d0 con 0x7f18b4038520 2024-04-24T23:49:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7f18c355c640 1 -- 172.21.15.16:0/3064651321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b40e2c30 con 0x7f18b4038520 2024-04-24T23:49:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7f18b1ffb640 1 -- 172.21.15.16:0/3064651321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18ac00cdc0 con 0x7f18b4038520 2024-04-24T23:49:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.501+0000 7f9313fff640 1 --2- 172.21.15.16:0/2481109477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9314143910 0x7f93140720a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f93140719c0 tx=0x7f930c02a6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79caa780e84d3376 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7f9312ffd640 1 -- 172.21.15.16:0/2481109477 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c036070 con 0x7f9314143910 2024-04-24T23:49:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7f9312ffd640 1 -- 172.21.15.16:0/2481109477 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f930c031600 con 0x7f9314143910 2024-04-24T23:49:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.502+0000 7f9312ffd640 1 -- 172.21.15.16:0/2481109477 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f930c031a10 con 0x7f9314143910 2024-04-24T23:49:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7f9319185640 1 -- 172.21.15.16:0/2481109477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9314143910 msgr2=0x7f93140720a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7f9319185640 1 --2- 172.21.15.16:0/2481109477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9314143910 0x7f93140720a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f93140719c0 tx=0x7f930c02a6b0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7fbf395f4640 1 --2- 172.21.15.16:0/889073818 >> v2:172.21.15.156:3300/0 conn(0x7fbf3415e020 0x7fbf340cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7fc4367fc640 1 -- 172.21.15.16:0/2799398445 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42403a070 con 0x7fc4300e1310 2024-04-24T23:49:21.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7fc4367fc640 1 -- 172.21.15.16:0/2799398445 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc42400b650 con 0x7fc4300e1310 2024-04-24T23:49:21.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7fc4367fc640 1 -- 172.21.15.16:0/2799398445 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42400ba20 con 0x7fc4300e1310 2024-04-24T23:49:21.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7f6cd0e35640 1 --2- 172.21.15.16:0/3692464151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb8004800 0x7f6cb8004c80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400a7c0 tx=0x7f6cb8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=953ba10426a3fbc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7fcb8759e640 1 Processor -- start 2024-04-24T23:49:21.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7fcb8759e640 1 -- start start 2024-04-24T23:49:21.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7f9319185640 1 -- 172.21.15.16:0/2481109477 shutdown_connections 2024-04-24T23:49:21.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7f9319185640 1 --2- 172.21.15.16:0/2481109477 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f931406e8c0 0x7f931415da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7f9319185640 1 --2- 172.21.15.16:0/2481109477 >> v2:172.21.15.16:3300/0 conn(0x7f93140725e0 0x7f931406e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7f9319185640 1 --2- 172.21.15.16:0/2481109477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9314143910 0x7f93140720a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7f9319185640 1 -- 172.21.15.16:0/2481109477 >> 172.21.15.16:0/2481109477 conn(0x7f931406b0e0 msgr2=0x7f931406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7f18c355c640 1 -- 172.21.15.16:0/3064651321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b404bf10 con 0x7f18b4038520 2024-04-24T23:49:21.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7f0af4c1b640 1 Processor -- start 2024-04-24T23:49:21.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.504+0000 7f8c157fa640 1 -- 172.21.15.16:0/4051269241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c3004f500 con 0x7f8c300b70f0 2024-04-24T23:49:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.504+0000 7f0af4c1b640 1 -- start start 2024-04-24T23:49:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.504+0000 7f18c355c640 1 -- 172.21.15.16:0/3064651321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b4005390 con 0x7f18b4038520 2024-04-24T23:49:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.504+0000 7fc43cfc5640 1 -- 172.21.15.16:0/2799398445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4300e1310 msgr2=0x7fc43003a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7fbf39df5640 1 -- 172.21.15.16:0/889073818 >> v2:172.21.15.156:3300/0 conn(0x7fbf3415e020 msgr2=0x7fbf340cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.503+0000 7fbf39df5640 1 --2- 172.21.15.16:0/889073818 >> v2:172.21.15.156:3300/0 conn(0x7fbf3415e020 0x7fbf340cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.504+0000 7fbf39df5640 1 -- 172.21.15.16:0/889073818 >> v1:172.21.15.16:6789/0 conn(0x7fbf340b2bb0 legacy=0x7fbf340cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.504+0000 7fbf39df5640 1 -- 172.21.15.16:0/889073818 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf3406c580 con 0x7fbf340d1fa0 2024-04-24T23:49:21.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.504+0000 7fc43cfc5640 1 --2- 172.21.15.16:0/2799398445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4300e1310 0x7fc43003a170 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc42400dc90 tx=0x7fc42402e500 comp rx=0 tx=0).stop 2024-04-24T23:49:21.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.504+0000 7f9319185640 1 -- 172.21.15.16:0/2481109477 shutdown_connections 2024-04-24T23:49:21.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.504+0000 7f9319185640 1 -- 172.21.15.16:0/2481109477 wait complete. 2024-04-24T23:49:21.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.504+0000 7fc43cfc5640 1 -- 172.21.15.16:0/2799398445 shutdown_connections 2024-04-24T23:49:21.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.504+0000 7fc43cfc5640 1 --2- 172.21.15.16:0/2799398445 >> v2:172.21.15.156:3300/0 conn(0x7fc43000ab40 0x7fc430009d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.504+0000 7fc43cfc5640 1 --2- 172.21.15.16:0/2799398445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4300e1310 0x7fc43003a170 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.504+0000 7fc43cfc5640 1 -- 172.21.15.16:0/2799398445 >> 172.21.15.16:0/2799398445 conn(0x7fc430056cd0 msgr2=0x7fc4300570e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7fcb8759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb880ce580 0x7fcb88159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7f6cc3fff640 1 -- 172.21.15.16:0/3692464151 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc400a3b0 con 0x7f6cb8004800 2024-04-24T23:49:21.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7fcb8759e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb88142a60 0x7fcb8813edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7f6cc3fff640 1 -- 172.21.15.16:0/3692464151 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc402f070 con 0x7f6cb8004800 2024-04-24T23:49:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7f6cc3fff640 1 -- 172.21.15.16:0/3692464151 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc402bc50 con 0x7f6cb8004800 2024-04-24T23:49:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7fcb8759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb881f8460 0x7fcb881f8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7f0af4c1b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ae80392d0 0x7f0ae80499e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7fcb8759e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8806d220 con 0x7fcb881f8460 2024-04-24T23:49:21.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7fcb8759e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcb880cd840 con 0x7fcb880ce580 2024-04-24T23:49:21.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7fcb8759e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fcb8806d3a0 con 0x7fcb88142a60 2024-04-24T23:49:21.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7f0af4c1b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ae804fa90 0x7f0ae804e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7fcb85d9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb88142a60 0x7fcb8813edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7fcb85d9b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb88142a60 0x7fcb8813edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:41542/0 (socket says 172.21.15.16:41542) 2024-04-24T23:49:21.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7fcb85d9b640 1 -- 172.21.15.16:0/1636463591 learned_addr learned my addr 172.21.15.16:0/1636463591 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7f0af4c1b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae80df420 0x7f0ae80d30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7f0af4c1b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae8052cc0 con 0x7f0ae80df420 2024-04-24T23:49:21.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7f0af4c1b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae8052b40 con 0x7f0ae80392d0 2024-04-24T23:49:21.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7f0af4c1b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae800aaa0 con 0x7f0ae804fa90 2024-04-24T23:49:21.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7fcb86d9d640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb881f8460 0x7fcb881f8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7f0aef7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ae80392d0 0x7f0ae80499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7f0aef7fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ae80392d0 0x7f0ae80499e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:34936/0 (socket says 172.21.15.16:34936) 2024-04-24T23:49:21.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7fc43cfc5640 1 -- 172.21.15.16:0/2799398445 shutdown_connections 2024-04-24T23:49:21.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7fc43cfc5640 1 -- 172.21.15.16:0/2799398445 wait complete. 2024-04-24T23:49:21.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.505+0000 7f0aef7fe640 1 -- 172.21.15.16:0/4120701126 learned_addr learned my addr 172.21.15.16:0/4120701126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f18b1ffb640 1 -- 172.21.15.16:0/3064651321 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f18ac01a070 con 0x7f18b4038520 2024-04-24T23:49:21.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7fcb85d9b640 1 -- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb880ce580 msgr2=0x7fcb88159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7fcb8659c640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb880ce580 0x7fcb88159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f6cd389a640 1 -- 172.21.15.16:0/3692464151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb8004800 msgr2=0x7f6cb8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f6cd389a640 1 --2- 172.21.15.16:0/3692464151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb8004800 0x7f6cb8004c80 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400a7c0 tx=0x7f6cb8005830 comp rx=0 tx=0).stop 2024-04-24T23:49:21.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f6cd1e37640 1 -- 172.21.15.16:0/3692464151 reap_dead start 2024-04-24T23:49:21.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7fcb85d9b640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb880ce580 0x7fcb88159430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7fcb85d9b640 1 -- 172.21.15.16:0/1636463591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb881f8460 msgr2=0x7fcb881f8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7fcb85d9b640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb881f8460 0x7fcb881f8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7fcb85d9b640 1 -- 172.21.15.16:0/1636463591 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb88159970 con 0x7fcb88142a60 2024-04-24T23:49:21.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f6cd389a640 1 -- 172.21.15.16:0/3692464151 shutdown_connections 2024-04-24T23:49:21.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f6cd389a640 1 -- 172.21.15.16:0/3692464151 >> 172.21.15.16:0/3692464151 conn(0x7f6ccc0cd770 msgr2=0x7f6ccc0cdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7fcb85d9b640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb88142a60 0x7fcb8813edb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcb80005e00 tx=0x7fcb80004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7fcb8659c640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb880ce580 0x7fcb88159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:21.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f18b1ffb640 1 --2- 172.21.15.16:0/3064651321 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f18a006bf30 0x7f18a006e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7fcb76ffd640 1 -- 172.21.15.16:0/1636463591 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80040020 con 0x7fcb88142a60 2024-04-24T23:49:21.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f0aef7fe640 1 -- 172.21.15.16:0/4120701126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ae804fa90 msgr2=0x7f0ae804e380 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:49:21.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f0aef7fe640 1 --2- 172.21.15.16:0/4120701126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ae804fa90 0x7f0ae804e380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f0aef7fe640 1 -- 172.21.15.16:0/4120701126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae80df420 msgr2=0x7f0ae80d30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f0aef7fe640 1 --2- 172.21.15.16:0/4120701126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae80df420 0x7f0ae80d30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7fcb76ffd640 1 -- 172.21.15.16:0/1636463591 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb80003ec0 con 0x7fcb88142a60 2024-04-24T23:49:21.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.507+0000 7fcb76ffd640 1 -- 172.21.15.16:0/1636463591 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb80038040 con 0x7fcb88142a60 2024-04-24T23:49:21.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f0aef7fe640 1 -- 172.21.15.16:0/4120701126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae804a0a0 con 0x7f0ae80392d0 2024-04-24T23:49:21.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f8c177fe640 1 -- 172.21.15.16:0/4051269241 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8c28005d10 con 0x7f8c300b70f0 2024-04-24T23:49:21.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f8c177fe640 1 --2- 172.21.15.16:0/4051269241 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8c1c06bf00 0x7f8c1c06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.507+0000 7f8c177fe640 1 -- 172.21.15.16:0/4051269241 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f8c2808c000 con 0x7f8c300b70f0 2024-04-24T23:49:21.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.507+0000 7f18b1ffb640 1 -- 172.21.15.16:0/3064651321 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f18ac08ba50 con 0x7f18b4038520 2024-04-24T23:49:21.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.507+0000 7fcb8759e640 1 -- 172.21.15.16:0/1636463591 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb881f8ef0 con 0x7fcb88142a60 2024-04-24T23:49:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f9319185640 1 Processor -- start 2024-04-24T23:49:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7f9319185640 1 -- start start 2024-04-24T23:49:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.506+0000 7fbf39df5640 1 --2- 172.21.15.16:0/889073818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf340d1fa0 0x7fbf340d23b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbf340d3de0 tx=0x7fbf2c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc4e4743bd1b54e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.507+0000 7f0aef7fe640 1 --2- 172.21.15.16:0/4120701126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ae80392d0 0x7f0ae80499e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400c430 tx=0x7f0ae400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.507+0000 7fcb8759e640 1 -- 172.21.15.16:0/1636463591 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb8813e5b0 con 0x7fcb88142a60 2024-04-24T23:49:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.507+0000 7f18c0af7640 1 --2- 172.21.15.16:0/3064651321 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f18a006bf30 0x7f18a006e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.507+0000 7f0adbfff640 1 -- 172.21.15.16:0/4120701126 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4011790 con 0x7f0ae80392d0 2024-04-24T23:49:21.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.507+0000 7f0af4c1b640 1 -- 172.21.15.16:0/4120701126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae804a380 con 0x7f0ae80392d0 2024-04-24T23:49:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.507+0000 7f0af4c1b640 1 -- 172.21.15.16:0/4120701126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae804a940 con 0x7f0ae80392d0 2024-04-24T23:49:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.507+0000 7f9319185640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f931406e8c0 0x7f93142003e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.507+0000 7f9319185640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93140725e0 0x7f93141574f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f0adbfff640 1 -- 172.21.15.16:0/4120701126 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ae4017070 con 0x7f0ae80392d0 2024-04-24T23:49:21.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f0adbfff640 1 -- 172.21.15.16:0/4120701126 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4011b70 con 0x7f0ae80392d0 2024-04-24T23:49:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f9319185640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9314143910 0x7f9314155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f9313fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f931406e8c0 0x7f93142003e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f9319185640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f931406ed30 con 0x7f93140725e0 2024-04-24T23:49:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f9319185640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f931406dd10 con 0x7f9314143910 2024-04-24T23:49:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f9319185640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f931406eeb0 con 0x7f931406e8c0 2024-04-24T23:49:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f9313fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f931406e8c0 0x7f93142003e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:41564/0 (socket says 172.21.15.16:41564) 2024-04-24T23:49:21.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f9313fff640 1 -- 172.21.15.16:0/983712529 learned_addr learned my addr 172.21.15.16:0/983712529 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f6cd389a640 1 -- 172.21.15.16:0/3692464151 shutdown_connections 2024-04-24T23:49:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f6cd389a640 1 -- 172.21.15.16:0/3692464151 wait complete. 2024-04-24T23:49:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7fc43cfc5640 1 Processor -- start 2024-04-24T23:49:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7fc43cfc5640 1 -- start start 2024-04-24T23:49:21.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f8c2e7fc640 1 --2- 172.21.15.16:0/4051269241 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8c1c06bf00 0x7f8c1c06e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.509+0000 7f18c0af7640 1 --2- 172.21.15.16:0/3064651321 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f18a006bf30 0x7f18a006e3f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f18b40e11f0 tx=0x7f18b8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f93137fe640 1 --2- 172.21.15.16:0/983712529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93140725e0 0x7f93141574f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f93137fe640 1 -- 172.21.15.16:0/983712529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f931406e8c0 msgr2=0x7f93142003e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f93137fe640 1 --2- 172.21.15.16:0/983712529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f931406e8c0 0x7f93142003e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f93137fe640 1 -- 172.21.15.16:0/983712529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9314143910 msgr2=0x7f9314155440 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:49:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f93137fe640 1 --2- 172.21.15.16:0/983712529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9314143910 0x7f9314155440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.508+0000 7f93137fe640 1 -- 172.21.15.16:0/983712529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f931415b720 con 0x7f93140725e0 2024-04-24T23:49:21.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.509+0000 7f9313fff640 1 --2- 172.21.15.16:0/983712529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f931406e8c0 0x7f93142003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:21.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.509+0000 7f93137fe640 1 --2- 172.21.15.16:0/983712529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93140725e0 0x7f93141574f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f930400e8a0 tx=0x7f930400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.509+0000 7f0adbfff640 1 -- 172.21.15.16:0/4120701126 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0ae4011d10 con 0x7f0ae80392d0 2024-04-24T23:49:21.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.509+0000 7fc43cfc5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc43000ab40 0x7fc4300d1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.509+0000 7fc43cfc5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4300516e0 0x7fc430050770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc43cfc5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4300e1310 0x7fc4300e16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.509+0000 7f0af4c1b640 1 -- 172.21.15.16:0/4120701126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ad0005290 con 0x7f0ae80392d0 2024-04-24T23:49:21.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.509+0000 7f8c2e7fc640 1 --2- 172.21.15.16:0/4051269241 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8c1c06bf00 0x7f8c1c06e3c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8c301f3cb0 tx=0x7f8c24009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.509+0000 7fcb74ff9640 1 -- 172.21.15.16:0/1636463591 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb8804f500 con 0x7fcb88142a60 2024-04-24T23:49:21.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc43cfc5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4300548a0 con 0x7fc4300516e0 2024-04-24T23:49:21.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc43cfc5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc4300555b0 con 0x7fc43000ab40 2024-04-24T23:49:21.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc4377fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc43000ab40 0x7fc4300d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc436ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4300516e0 0x7fc430050770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc43cfc5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fc430054a20 con 0x7fc4300e1310 2024-04-24T23:49:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc4377fe640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc43000ab40 0x7fc4300d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:34964/0 (socket says 172.21.15.16:34964) 2024-04-24T23:49:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc436ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4300516e0 0x7fc430050770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48528/0 (socket says 172.21.15.16:48528) 2024-04-24T23:49:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc4377fe640 1 -- 172.21.15.16:0/4224237314 learned_addr learned my addr 172.21.15.16:0/4224237314 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc437fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4300e1310 0x7fc4300e16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7f6cd389a640 1 Processor -- start 2024-04-24T23:49:21.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7f0adbfff640 1 --2- 172.21.15.16:0/4120701126 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0ad406be80 0x7f0ad406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7f6cd389a640 1 -- start start 2024-04-24T23:49:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc4377fe640 1 -- 172.21.15.16:0/4224237314 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4300e1310 msgr2=0x7fc4300e16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7f0adbfff640 1 -- 172.21.15.16:0/4120701126 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f0ae408afc0 con 0x7f0ae80392d0 2024-04-24T23:49:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc4377fe640 1 --2- 172.21.15.16:0/4224237314 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4300e1310 0x7fc4300e16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc4377fe640 1 -- 172.21.15.16:0/4224237314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4300516e0 msgr2=0x7fc430050770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc4377fe640 1 --2- 172.21.15.16:0/4224237314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4300516e0 0x7fc430050770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fc4377fe640 1 -- 172.21.15.16:0/4224237314 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4300d1650 con 0x7fc43000ab40 2024-04-24T23:49:21.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7f0aeeffd640 1 --2- 172.21.15.16:0/4120701126 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0ad406be80 0x7f0ad406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.510+0000 7fcb76ffd640 1 -- 172.21.15.16:0/1636463591 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcb80043020 con 0x7fcb88142a60 2024-04-24T23:49:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.511+0000 7fc4377fe640 1 --2- 172.21.15.16:0/4224237314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc43000ab40 0x7fc4300d1110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc42400dc90 tx=0x7fc4240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.511+0000 7fcb76ffd640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcb6806bf30 0x7fcb6806e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.511+0000 7fcb76ffd640 1 -- 172.21.15.16:0/1636463591 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fcb800ab7d0 con 0x7fcb88142a60 2024-04-24T23:49:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.511+0000 7fcb8659c640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcb6806bf30 0x7fcb6806e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.511+0000 7f0aeeffd640 1 --2- 172.21.15.16:0/4120701126 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0ad406be80 0x7f0ad406e340 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00b860 tx=0x7f0adc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.511+0000 7f6cd389a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb8004800 0x7f6ccc1406a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.511+0000 7f6cd389a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ccc15e610 0x7f6ccc142f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.511+0000 7f6cd389a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ccc15da60 0x7f6ccc15dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.511+0000 7f6cd389a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6ccc1634c0 con 0x7f6ccc15da60 2024-04-24T23:49:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.511+0000 7f6cd389a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6ccc163340 con 0x7f6ccc15e610 2024-04-24T23:49:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.512+0000 7f6cd389a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6ccc163640 con 0x7f6cb8004800 2024-04-24T23:49:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.512+0000 7f6cd0e35640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ccc15e610 0x7f6ccc142f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.512+0000 7f6cd0e35640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ccc15e610 0x7f6ccc142f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:34980/0 (socket says 172.21.15.16:34980) 2024-04-24T23:49:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.512+0000 7f6cd0e35640 1 -- 172.21.15.16:0/425859501 learned_addr learned my addr 172.21.15.16:0/425859501 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.512+0000 7f6cd1636640 1 --2- 172.21.15.16:0/425859501 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb8004800 0x7f6ccc1406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.512+0000 7f6cd1e37640 1 --2- 172.21.15.16:0/425859501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ccc15da60 0x7f6ccc15dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.512+0000 7fcb8659c640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcb6806bf30 0x7fcb6806e3f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcb78006fd0 tx=0x7fcb78008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.512+0000 7f6cd0e35640 1 -- 172.21.15.16:0/425859501 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb8004800 msgr2=0x7f6ccc1406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.512+0000 7f6cd0e35640 1 --2- 172.21.15.16:0/425859501 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb8004800 0x7f6ccc1406a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.512+0000 7f6cd0e35640 1 -- 172.21.15.16:0/425859501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ccc15da60 msgr2=0x7f6ccc15dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.512+0000 7f6cd0e35640 1 --2- 172.21.15.16:0/425859501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ccc15da60 0x7f6ccc15dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.512+0000 7f6cd0e35640 1 -- 172.21.15.16:0/425859501 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ccc15e420 con 0x7f6ccc15e610 2024-04-24T23:49:21.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.512+0000 7f6cd0e35640 1 --2- 172.21.15.16:0/425859501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ccc15e610 0x7f6ccc142f40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400a8e0 tx=0x7f6ccc166560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.513+0000 7f18b1ffb640 1 -- 172.21.15.16:0/3064651321 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f18ac05a010 con 0x7f18b4038520 2024-04-24T23:49:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.515+0000 7f6cc1ffb640 1 -- 172.21.15.16:0/425859501 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc400a3b0 con 0x7f6ccc15e610 2024-04-24T23:49:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.515+0000 7f6cc1ffb640 1 -- 172.21.15.16:0/425859501 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6cc4004d70 con 0x7f6ccc15e610 2024-04-24T23:49:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.515+0000 7f6cc1ffb640 1 -- 172.21.15.16:0/425859501 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cc402bd60 con 0x7f6ccc15e610 2024-04-24T23:49:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.515+0000 7f6cd389a640 1 -- 172.21.15.16:0/425859501 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ccc1411e0 con 0x7f6ccc15e610 2024-04-24T23:49:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.515+0000 7fc423fff640 1 -- 172.21.15.16:0/4224237314 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc424043020 con 0x7fc43000ab40 2024-04-24T23:49:21.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.515+0000 7fc423fff640 1 -- 172.21.15.16:0/4224237314 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc42403f610 con 0x7fc43000ab40 2024-04-24T23:49:21.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.515+0000 7fc423fff640 1 -- 172.21.15.16:0/4224237314 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42403a040 con 0x7fc43000ab40 2024-04-24T23:49:21.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.515+0000 7fc43cfc5640 1 -- 172.21.15.16:0/4224237314 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc43004f6d0 con 0x7fc43000ab40 2024-04-24T23:49:21.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.515+0000 7fc43cfc5640 1 -- 172.21.15.16:0/4224237314 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc43004fc90 con 0x7fc43000ab40 2024-04-24T23:49:21.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.515+0000 7f0adbfff640 1 -- 172.21.15.16:0/4120701126 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0ae4059610 con 0x7f0ae80392d0 2024-04-24T23:49:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.516+0000 7f6cd389a640 1 -- 172.21.15.16:0/425859501 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ccc1469a0 con 0x7f6ccc15e610 2024-04-24T23:49:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.517+0000 7fcb76ffd640 1 -- 172.21.15.16:0/1636463591 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcb8004a030 con 0x7fcb88142a60 2024-04-24T23:49:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.517+0000 7fc43cfc5640 1 -- 172.21.15.16:0/4224237314 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4300057e0 con 0x7fc43000ab40 2024-04-24T23:49:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.518+0000 7fbf2bfff640 1 -- 172.21.15.16:0/889073818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c036070 con 0x7fbf340d1fa0 2024-04-24T23:49:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.518+0000 7fbf2bfff640 1 -- 172.21.15.16:0/889073818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf2c031600 con 0x7fbf340d1fa0 2024-04-24T23:49:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.518+0000 7fbf2bfff640 1 -- 172.21.15.16:0/889073818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c0319d0 con 0x7fbf340d1fa0 2024-04-24T23:49:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.518+0000 7f8c177fe640 1 -- 172.21.15.16:0/4051269241 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8c28022020 con 0x7f8c300b70f0 2024-04-24T23:49:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.518+0000 7f9310ff9640 1 -- 172.21.15.16:0/983712529 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9304017070 con 0x7f93140725e0 2024-04-24T23:49:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.518+0000 7f9310ff9640 1 -- 172.21.15.16:0/983712529 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f930400c940 con 0x7f93140725e0 2024-04-24T23:49:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.518+0000 7fc423fff640 1 -- 172.21.15.16:0/4224237314 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc424046020 con 0x7fc43000ab40 2024-04-24T23:49:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.518+0000 7fc423fff640 1 --2- 172.21.15.16:0/4224237314 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc41c06be30 0x7fc41c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.519+0000 7fc423fff640 1 -- 172.21.15.16:0/4224237314 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc424076980 con 0x7fc43000ab40 2024-04-24T23:49:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.518+0000 7f9319185640 1 -- 172.21.15.16:0/983712529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93140b5750 con 0x7f93140725e0 2024-04-24T23:49:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.518+0000 7f9319185640 1 -- 172.21.15.16:0/983712529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93140b5d10 con 0x7f93140725e0 2024-04-24T23:49:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.519+0000 7f9310ff9640 1 -- 172.21.15.16:0/983712529 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9304009e60 con 0x7f93140725e0 2024-04-24T23:49:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.519+0000 7fbf3b858640 1 -- 172.21.15.16:0/889073818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf340d1fa0 msgr2=0x7fbf340d23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.519+0000 7fbf3b858640 1 --2- 172.21.15.16:0/889073818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf340d1fa0 0x7fbf340d23b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbf340d3de0 tx=0x7fbf2c02a660 comp rx=0 tx=0).stop 2024-04-24T23:49:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.519+0000 7fbf3b858640 1 -- 172.21.15.16:0/889073818 shutdown_connections 2024-04-24T23:49:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.519+0000 7fbf3b858640 1 --2- 172.21.15.16:0/889073818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf340d1fa0 0x7fbf340d23b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.519+0000 7fbf3b858640 1 --2- 172.21.15.16:0/889073818 >> v2:172.21.15.156:3300/0 conn(0x7fbf3415e020 0x7fbf340cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.519+0000 7fbf3b858640 1 -- 172.21.15.16:0/889073818 >> 172.21.15.16:0/889073818 conn(0x7fbf340d3e10 msgr2=0x7fbf340d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.519+0000 7fbf3b858640 1 -- 172.21.15.16:0/889073818 shutdown_connections 2024-04-24T23:49:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.519+0000 7fbf3b858640 1 -- 172.21.15.16:0/889073818 wait complete. 2024-04-24T23:49:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.520+0000 7f9319185640 1 -- 172.21.15.16:0/983712529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93140b5f10 con 0x7f93140725e0 2024-04-24T23:49:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.521+0000 7fc436ffd640 1 --2- 172.21.15.16:0/4224237314 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc41c06be30 0x7fc41c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.517+0000 7f6cd389a640 1 -- 172.21.15.16:0/425859501 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ccc06b0e0 con 0x7f6ccc15e610 2024-04-24T23:49:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.519+0000 7f6cc1ffb640 1 -- 172.21.15.16:0/425859501 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6cc4034470 con 0x7f6ccc15e610 2024-04-24T23:49:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.520+0000 7f6cc1ffb640 1 --2- 172.21.15.16:0/425859501 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6cb006bf30 0x7f6cb006e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.520+0000 7f6cc1ffb640 1 -- 172.21.15.16:0/425859501 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f6cc40abe70 con 0x7f6ccc15e610 2024-04-24T23:49:21.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.521+0000 7f6cd1636640 1 --2- 172.21.15.16:0/425859501 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6cb006bf30 0x7f6cb006e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.521+0000 7fc436ffd640 1 --2- 172.21.15.16:0/4224237314 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc41c06be30 0x7fc41c06e2f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc430052680 tx=0x7fc42c009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.521+0000 7fbf3b858640 1 Processor -- start 2024-04-24T23:49:21.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.521+0000 7fbf3b858640 1 -- start start 2024-04-24T23:49:21.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.521+0000 7f9310ff9640 1 -- 172.21.15.16:0/983712529 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f930401a070 con 0x7f93140725e0 2024-04-24T23:49:21.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.522+0000 7f6cd1636640 1 --2- 172.21.15.16:0/425859501 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6cb006bf30 0x7f6cb006e3f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc073700 tx=0x7f6cbc009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.522+0000 7f9310ff9640 1 --2- 172.21.15.16:0/983712529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f92f406beb0 0x7f92f406e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.522+0000 7f9310ff9640 1 -- 172.21.15.16:0/983712529 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f930408ade0 con 0x7f93140725e0 2024-04-24T23:49:21.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.522+0000 7f9313fff640 1 --2- 172.21.15.16:0/983712529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f92f406beb0 0x7f92f406e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.523+0000 7fbf3b858640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf340b2bb0 0x7fbf340b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.523+0000 7fbf3b858640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf340d1fa0 0x7fbf340b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.523+0000 7fbf3b858640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3415e020 0x7fbf340c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.523+0000 7fbf395f4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf340b2bb0 0x7fbf340b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.523+0000 7fbf39df5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3415e020 0x7fbf340c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.523+0000 7fbf39df5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3415e020 0x7fbf340c76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48544/0 (socket says 172.21.15.16:48544) 2024-04-24T23:49:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.523+0000 7fbf39df5640 1 -- 172.21.15.16:0/2419320339 learned_addr learned my addr 172.21.15.16:0/2419320339 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.523+0000 7fbf3b858640 1 -- 172.21.15.16:0/2419320339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbf3415e400 con 0x7fbf3415e020 2024-04-24T23:49:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.523+0000 7fbf3b858640 1 -- 172.21.15.16:0/2419320339 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbf340d1a80 con 0x7fbf340b2bb0 2024-04-24T23:49:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.523+0000 7fbf3b858640 1 -- 172.21.15.16:0/2419320339 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fbf340cfec0 con 0x7fbf340d1fa0 2024-04-24T23:49:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.524+0000 7fbf38df3640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf340d1fa0 0x7fbf340b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.524+0000 7f9313fff640 1 --2- 172.21.15.16:0/983712529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f92f406beb0 0x7f92f406e370 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f930c00c090 tx=0x7f930c009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.524+0000 7f6cc1ffb640 1 -- 172.21.15.16:0/425859501 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6cc4046060 con 0x7f6ccc15e610 2024-04-24T23:49:21.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.524+0000 7fbf39df5640 1 -- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf340d1fa0 msgr2=0x7fbf340b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.524+0000 7fbf39df5640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf340d1fa0 0x7fbf340b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.524+0000 7fbf39df5640 1 -- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf340b2bb0 msgr2=0x7fbf340b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.524+0000 7fbf39df5640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf340b2bb0 0x7fbf340b4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.524+0000 7fbf39df5640 1 -- 172.21.15.16:0/2419320339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf341f7f20 con 0x7fbf3415e020 2024-04-24T23:49:21.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.524+0000 7fbf39df5640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3415e020 0x7fbf340c76c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c02ac50 tx=0x7fbf2c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.527+0000 7fbf29ffb640 1 -- 172.21.15.16:0/2419320339 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c041020 con 0x7fbf3415e020 2024-04-24T23:49:21.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.527+0000 7fbf29ffb640 1 -- 172.21.15.16:0/2419320339 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf2c008710 con 0x7fbf3415e020 2024-04-24T23:49:21.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.527+0000 7fbf29ffb640 1 -- 172.21.15.16:0/2419320339 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c036040 con 0x7fbf3415e020 2024-04-24T23:49:21.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.528+0000 7fbf3b858640 1 -- 172.21.15.16:0/2419320339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf340ca4d0 con 0x7fbf3415e020 2024-04-24T23:49:21.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.528+0000 7fbf3b858640 1 -- 172.21.15.16:0/2419320339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf341f8480 con 0x7fbf3415e020 2024-04-24T23:49:21.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.529+0000 7fc423fff640 1 -- 172.21.15.16:0/4224237314 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc4300d1650 con 0x7fc43000ab40 2024-04-24T23:49:21.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.529+0000 7fbf3b858640 1 -- 172.21.15.16:0/2419320339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf3406dfe0 con 0x7fbf3415e020 2024-04-24T23:49:21.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.527+0000 7f9310ff9640 1 -- 172.21.15.16:0/983712529 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f93040595c0 con 0x7f93140725e0 2024-04-24T23:49:21.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.530+0000 7fbf38df3640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf340d1fa0 0x7fbf340b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:21.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.532+0000 7fbf395f4640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf340b2bb0 0x7fbf340b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:21.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.541+0000 7fbf29ffb640 1 -- 172.21.15.16:0/2419320339 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbf2c044020 con 0x7fbf3415e020 2024-04-24T23:49:21.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.541+0000 7fbf29ffb640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbf1806bf00 0x7fbf1806e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:21.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.541+0000 7fbf29ffb640 1 -- 172.21.15.16:0/2419320339 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbf2c0ac090 con 0x7fbf3415e020 2024-04-24T23:49:21.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.542+0000 7fbf29ffb640 1 -- 172.21.15.16:0/2419320339 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbf2c04b030 con 0x7fbf3415e020 2024-04-24T23:49:21.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.555+0000 7fbf395f4640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbf1806bf00 0x7fbf1806e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:21.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.564+0000 7fbf395f4640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbf1806bf00 0x7fbf1806e3c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbf340c96b0 tx=0x7fbf30008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.933+0000 7f18c355c640 1 -- 172.21.15.16:0/3064651321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f18b40074f0 con 0x7f18b4038520 2024-04-24T23:49:21.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.934+0000 7f18b1ffb640 1 -- 172.21.15.16:0/3064651321 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f18ac059910 con 0x7f18b4038520 2024-04-24T23:49:21.934 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2024-04-24T23:49:21.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.937+0000 7f18a77fe640 1 -- 172.21.15.16:0/3064651321 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f18a006bf30 msgr2=0x7f18a006e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.937+0000 7f18a77fe640 1 --2- 172.21.15.16:0/3064651321 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f18a006bf30 0x7f18a006e3f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f18b40e11f0 tx=0x7f18b8009040 comp rx=0 tx=0).stop 2024-04-24T23:49:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.937+0000 7f18a77fe640 1 -- 172.21.15.16:0/3064651321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18b4038520 msgr2=0x7f18b4174740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.937+0000 7f18a77fe640 1 --2- 172.21.15.16:0/3064651321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18b4038520 0x7f18b4174740 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f18ac00e9e0 tx=0x7f18ac00eea0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.938+0000 7f18a77fe640 1 -- 172.21.15.16:0/3064651321 shutdown_connections 2024-04-24T23:49:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.938+0000 7f18a77fe640 1 --2- 172.21.15.16:0/3064651321 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f18a006bf30 0x7f18a006e3f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.938+0000 7f18a77fe640 1 --2- 172.21.15.16:0/3064651321 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f18b4054200 0x7f18b40ce6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.938+0000 7f18a77fe640 1 --2- 172.21.15.16:0/3064651321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18b4050740 0x7f18b40e1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.938+0000 7f18a77fe640 1 --2- 172.21.15.16:0/3064651321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18b4038520 0x7f18b4174740 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.938+0000 7f18a77fe640 1 -- 172.21.15.16:0/3064651321 >> 172.21.15.16:0/3064651321 conn(0x7f18b4009fb0 msgr2=0x7f18b400a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.938+0000 7f18a77fe640 1 -- 172.21.15.16:0/3064651321 shutdown_connections 2024-04-24T23:49:21.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.939+0000 7f18a77fe640 1 -- 172.21.15.16:0/3064651321 wait complete. 2024-04-24T23:49:21.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.948+0000 7f0af4c1b640 1 -- 172.21.15.16:0/4120701126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f0ad0005580 con 0x7f0ae80392d0 2024-04-24T23:49:21.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.948+0000 7f0adbfff640 1 -- 172.21.15.16:0/4120701126 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0ae4058f10 con 0x7f0ae80392d0 2024-04-24T23:49:21.948 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2024-04-24T23:49:21.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.952+0000 7f0ad9ffb640 1 -- 172.21.15.16:0/4120701126 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0ad406be80 msgr2=0x7f0ad406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.952+0000 7f0ad9ffb640 1 --2- 172.21.15.16:0/4120701126 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0ad406be80 0x7f0ad406e340 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00b860 tx=0x7f0adc008040 comp rx=0 tx=0).stop 2024-04-24T23:49:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.952+0000 7f0ad9ffb640 1 -- 172.21.15.16:0/4120701126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ae80392d0 msgr2=0x7f0ae80499e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.952+0000 7f0ad9ffb640 1 --2- 172.21.15.16:0/4120701126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ae80392d0 0x7f0ae80499e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0ae400c430 tx=0x7f0ae400c8f0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.953+0000 7f0ad9ffb640 1 -- 172.21.15.16:0/4120701126 shutdown_connections 2024-04-24T23:49:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.953+0000 7f0ad9ffb640 1 --2- 172.21.15.16:0/4120701126 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0ad406be80 0x7f0ad406e340 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.953+0000 7f0ad9ffb640 1 --2- 172.21.15.16:0/4120701126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0ae80df420 0x7f0ae80d30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.953+0000 7f0ad9ffb640 1 --2- 172.21.15.16:0/4120701126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0ae804fa90 0x7f0ae804e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.953+0000 7f0ad9ffb640 1 --2- 172.21.15.16:0/4120701126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0ae80392d0 0x7f0ae80499e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.953+0000 7f0ad9ffb640 1 -- 172.21.15.16:0/4120701126 >> 172.21.15.16:0/4120701126 conn(0x7f0ae8009fb0 msgr2=0x7f0ae80dc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.953+0000 7f0ad9ffb640 1 -- 172.21.15.16:0/4120701126 shutdown_connections 2024-04-24T23:49:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.953+0000 7f0ad9ffb640 1 -- 172.21.15.16:0/4120701126 wait complete. 2024-04-24T23:49:21.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.954+0000 7fcb74ff9640 1 -- 172.21.15.16:0/1636463591 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fcb880b0b90 con 0x7fcb88142a60 2024-04-24T23:49:21.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.954+0000 7fcb76ffd640 1 -- 172.21.15.16:0/1636463591 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fcb88159970 con 0x7fcb88142a60 2024-04-24T23:49:21.955 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2024-04-24T23:49:21.956 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.6 2024-04-24T23:49:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.956+0000 7f6cd389a640 1 -- 172.21.15.16:0/425859501 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f6ccc04f5e0 con 0x7f6ccc15e610 2024-04-24T23:49:21.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.957+0000 7f6cc1ffb640 1 -- 172.21.15.16:0/425859501 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6cc4079de0 con 0x7f6ccc15e610 2024-04-24T23:49:21.957 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2024-04-24T23:49:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.958+0000 7fc43cfc5640 1 -- 172.21.15.16:0/4224237314 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc4300578f0 con 0x7fc43000ab40 2024-04-24T23:49:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.958+0000 7fcb74ff9640 1 -- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcb6806bf30 msgr2=0x7fcb6806e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.958+0000 7fcb74ff9640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcb6806bf30 0x7fcb6806e3f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcb78006fd0 tx=0x7fcb78008040 comp rx=0 tx=0).stop 2024-04-24T23:49:21.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.958+0000 7fcb74ff9640 1 -- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb88142a60 msgr2=0x7fcb8813edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.958+0000 7fcb74ff9640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb88142a60 0x7fcb8813edb0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcb80005e00 tx=0x7fcb80004a40 comp rx=0 tx=0).stop 2024-04-24T23:49:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.958+0000 7fcb74ff9640 1 -- 172.21.15.16:0/1636463591 shutdown_connections 2024-04-24T23:49:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.958+0000 7fcb74ff9640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcb6806bf30 0x7fcb6806e3f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.958+0000 7fcb74ff9640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb881f8460 0x7fcb881f8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.958+0000 7fcb74ff9640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcb88142a60 0x7fcb8813edb0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.958+0000 7fcb74ff9640 1 --2- 172.21.15.16:0/1636463591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcb880ce580 0x7fcb88159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.958+0000 7fcb74ff9640 1 -- 172.21.15.16:0/1636463591 >> 172.21.15.16:0/1636463591 conn(0x7fcb880af700 msgr2=0x7fcb880717c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.959+0000 7fcb74ff9640 1 -- 172.21.15.16:0/1636463591 shutdown_connections 2024-04-24T23:49:21.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.959+0000 7fc423fff640 1 -- 172.21.15.16:0/4224237314 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc4300d1650 con 0x7fc43000ab40 2024-04-24T23:49:21.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.959+0000 7fcb74ff9640 1 -- 172.21.15.16:0/1636463591 wait complete. 2024-04-24T23:49:21.960 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2024-04-24T23:49:21.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.960+0000 7f6cd389a640 1 -- 172.21.15.16:0/425859501 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6cb006bf30 msgr2=0x7f6cb006e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.960+0000 7f6cd389a640 1 --2- 172.21.15.16:0/425859501 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6cb006bf30 0x7f6cb006e3f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc073700 tx=0x7f6cbc009190 comp rx=0 tx=0).stop 2024-04-24T23:49:21.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.960+0000 7f6cd389a640 1 -- 172.21.15.16:0/425859501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ccc15e610 msgr2=0x7f6ccc142f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.960+0000 7f6cd389a640 1 --2- 172.21.15.16:0/425859501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ccc15e610 0x7f6ccc142f40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400a8e0 tx=0x7f6ccc166560 comp rx=0 tx=0).stop 2024-04-24T23:49:21.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.960+0000 7f6cd389a640 1 -- 172.21.15.16:0/425859501 shutdown_connections 2024-04-24T23:49:21.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.960+0000 7f6cd389a640 1 --2- 172.21.15.16:0/425859501 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6cb006bf30 0x7f6cb006e3f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.960+0000 7f6cd389a640 1 --2- 172.21.15.16:0/425859501 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ccc15da60 0x7f6ccc15dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.960+0000 7f6cd389a640 1 --2- 172.21.15.16:0/425859501 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ccc15e610 0x7f6ccc142f40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.960+0000 7f6cd389a640 1 --2- 172.21.15.16:0/425859501 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb8004800 0x7f6ccc1406a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.960+0000 7f6cd389a640 1 -- 172.21.15.16:0/425859501 >> 172.21.15.16:0/425859501 conn(0x7f6ccc0cd770 msgr2=0x7f6ccc0b15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.961+0000 7f6cd389a640 1 -- 172.21.15.16:0/425859501 shutdown_connections 2024-04-24T23:49:21.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.961+0000 7f6cd389a640 1 -- 172.21.15.16:0/425859501 wait complete. 2024-04-24T23:49:21.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.962+0000 7fc43cfc5640 1 -- 172.21.15.16:0/4224237314 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc41c06be30 msgr2=0x7fc41c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.962+0000 7fc43cfc5640 1 --2- 172.21.15.16:0/4224237314 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc41c06be30 0x7fc41c06e2f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc430052680 tx=0x7fc42c009190 comp rx=0 tx=0).stop 2024-04-24T23:49:21.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.962+0000 7fc43cfc5640 1 -- 172.21.15.16:0/4224237314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc43000ab40 msgr2=0x7fc4300d1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.962+0000 7fc43cfc5640 1 --2- 172.21.15.16:0/4224237314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc43000ab40 0x7fc4300d1110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc42400dc90 tx=0x7fc4240046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.962+0000 7fc43cfc5640 1 -- 172.21.15.16:0/4224237314 shutdown_connections 2024-04-24T23:49:21.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.962+0000 7fc43cfc5640 1 --2- 172.21.15.16:0/4224237314 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc41c06be30 0x7fc41c06e2f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.962+0000 7fc43cfc5640 1 --2- 172.21.15.16:0/4224237314 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4300e1310 0x7fc4300e16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.962+0000 7fc43cfc5640 1 --2- 172.21.15.16:0/4224237314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4300516e0 0x7fc430050770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.962+0000 7fc43cfc5640 1 --2- 172.21.15.16:0/4224237314 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc43000ab40 0x7fc4300d1110 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.962+0000 7fc43cfc5640 1 -- 172.21.15.16:0/4224237314 >> 172.21.15.16:0/4224237314 conn(0x7fc430056cd0 msgr2=0x7fc43003b1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.962+0000 7fc43cfc5640 1 -- 172.21.15.16:0/4224237314 shutdown_connections 2024-04-24T23:49:21.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.962+0000 7fc43cfc5640 1 -- 172.21.15.16:0/4224237314 wait complete. 2024-04-24T23:49:21.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.964+0000 7f8c157fa640 1 -- 172.21.15.16:0/4051269241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8c300b31f0 con 0x7f8c300b70f0 2024-04-24T23:49:21.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.964+0000 7f8c177fe640 1 -- 172.21.15.16:0/4051269241 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8c2805a650 con 0x7f8c300b70f0 2024-04-24T23:49:21.965 INFO:teuthology.orchestra.run.smithi016.stdout:55834574858 2024-04-24T23:49:21.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.967+0000 7f9319185640 1 -- 172.21.15.16:0/983712529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f931415b8f0 con 0x7f93140725e0 2024-04-24T23:49:21.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.967+0000 7f9310ff9640 1 -- 172.21.15.16:0/983712529 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9304058ec0 con 0x7f93140725e0 2024-04-24T23:49:21.968 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2024-04-24T23:49:21.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.968+0000 7f8c157fa640 1 -- 172.21.15.16:0/4051269241 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8c1c06bf00 msgr2=0x7f8c1c06e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.968+0000 7f8c157fa640 1 --2- 172.21.15.16:0/4051269241 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8c1c06bf00 0x7f8c1c06e3c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8c301f3cb0 tx=0x7f8c24009040 comp rx=0 tx=0).stop 2024-04-24T23:49:21.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.968+0000 7f8c157fa640 1 -- 172.21.15.16:0/4051269241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c300b70f0 msgr2=0x7f8c301f3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.968+0000 7f8c157fa640 1 --2- 172.21.15.16:0/4051269241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c300b70f0 0x7f8c301f3d20 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8c2800eec0 tx=0x7f8c2800b480 comp rx=0 tx=0).stop 2024-04-24T23:49:21.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.968+0000 7f8c157fa640 1 -- 172.21.15.16:0/4051269241 shutdown_connections 2024-04-24T23:49:21.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.968+0000 7f8c157fa640 1 --2- 172.21.15.16:0/4051269241 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8c1c06bf00 0x7f8c1c06e3c0 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8c301f3cb0 tx=0x7f8c24009040 comp rx=0 tx=0).stop 2024-04-24T23:49:21.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.968+0000 7f8c157fa640 1 --2- 172.21.15.16:0/4051269241 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8c30166420 0x7f8c30141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.968+0000 7f8c157fa640 1 --2- 172.21.15.16:0/4051269241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c30148580 0x7f8c30204c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.968+0000 7f8c157fa640 1 --2- 172.21.15.16:0/4051269241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c300b70f0 0x7f8c301f3d20 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.968+0000 7f8c157fa640 1 -- 172.21.15.16:0/4051269241 >> 172.21.15.16:0/4051269241 conn(0x7f8c3006b500 msgr2=0x7f8c30145ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.968+0000 7f8c157fa640 1 -- 172.21.15.16:0/4051269241 shutdown_connections 2024-04-24T23:49:21.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.968+0000 7f8c157fa640 1 -- 172.21.15.16:0/4051269241 wait complete. 2024-04-24T23:49:21.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.971+0000 7f9319185640 1 -- 172.21.15.16:0/983712529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f92f406beb0 msgr2=0x7f92f406e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.971+0000 7f9319185640 1 --2- 172.21.15.16:0/983712529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f92f406beb0 0x7f92f406e370 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f930c00c090 tx=0x7f930c009000 comp rx=0 tx=0).stop 2024-04-24T23:49:21.972 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.3 2024-04-24T23:49:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.971+0000 7f9319185640 1 -- 172.21.15.16:0/983712529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93140725e0 msgr2=0x7f93141574f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.971+0000 7f9319185640 1 --2- 172.21.15.16:0/983712529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93140725e0 0x7f93141574f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f930400e8a0 tx=0x7f930400ed60 comp rx=0 tx=0).stop 2024-04-24T23:49:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.971+0000 7f9319185640 1 -- 172.21.15.16:0/983712529 shutdown_connections 2024-04-24T23:49:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.971+0000 7f9319185640 1 --2- 172.21.15.16:0/983712529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f92f406beb0 0x7f92f406e370 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.971+0000 7f9319185640 1 --2- 172.21.15.16:0/983712529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9314143910 0x7f9314155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.971+0000 7f9319185640 1 --2- 172.21.15.16:0/983712529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93140725e0 0x7f93141574f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.971+0000 7f9319185640 1 --2- 172.21.15.16:0/983712529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f931406e8c0 0x7f93142003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.971+0000 7f9319185640 1 -- 172.21.15.16:0/983712529 >> 172.21.15.16:0/983712529 conn(0x7f931406b0e0 msgr2=0x7f931406f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.972+0000 7f9319185640 1 -- 172.21.15.16:0/983712529 shutdown_connections 2024-04-24T23:49:21.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.972+0000 7f9319185640 1 -- 172.21.15.16:0/983712529 wait complete. 2024-04-24T23:49:21.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.973+0000 7fbf3b858640 1 -- 172.21.15.16:0/2419320339 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fbf3404f5e0 con 0x7fbf3415e020 2024-04-24T23:49:21.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.974+0000 7fbf29ffb640 1 -- 172.21.15.16:0/2419320339 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbf341f7f20 con 0x7fbf3415e020 2024-04-24T23:49:21.974 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2024-04-24T23:49:21.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.977+0000 7fbf3b858640 1 -- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbf1806bf00 msgr2=0x7fbf1806e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.977+0000 7fbf3b858640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbf1806bf00 0x7fbf1806e3c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbf340c96b0 tx=0x7fbf30008040 comp rx=0 tx=0).stop 2024-04-24T23:49:21.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.977+0000 7fbf3b858640 1 -- 172.21.15.16:0/2419320339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3415e020 msgr2=0x7fbf340c76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:21.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.977+0000 7fbf3b858640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3415e020 0x7fbf340c76c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c02ac50 tx=0x7fbf2c004a40 comp rx=0 tx=0).stop 2024-04-24T23:49:21.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.977+0000 7fbf3b858640 1 -- 172.21.15.16:0/2419320339 shutdown_connections 2024-04-24T23:49:21.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.977+0000 7fbf3b858640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbf1806bf00 0x7fbf1806e3c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.977+0000 7fbf3b858640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf3415e020 0x7fbf340c76c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.977+0000 7fbf3b858640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf340d1fa0 0x7fbf340b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.977+0000 7fbf3b858640 1 --2- 172.21.15.16:0/2419320339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf340b2bb0 0x7fbf340b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:21.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.977+0000 7fbf3b858640 1 -- 172.21.15.16:0/2419320339 >> 172.21.15.16:0/2419320339 conn(0x7fbf340d3e10 msgr2=0x7fbf3415d3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:21.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.978+0000 7fbf3b858640 1 -- 172.21.15.16:0/2419320339 shutdown_connections 2024-04-24T23:49:21.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:21.978+0000 7fbf3b858640 1 -- 172.21.15.16:0/2419320339 wait complete. 2024-04-24T23:49:21.980 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.1 2024-04-24T23:49:21.981 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.4 2024-04-24T23:49:21.982 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.5 2024-04-24T23:49:21.985 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574858 for osd.2 2024-04-24T23:49:21.991 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.0 2024-04-24T23:49:21.992 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.7 2024-04-24T23:49:22.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-24T23:49:22.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-04-24T23:49:22.981 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-04-24T23:49:22.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-04-24T23:49:22.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-04-24T23:49:22.986 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-04-24T23:49:22.992 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-04-24T23:49:22.994 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-04-24T23:49:23.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.123+0000 7fbcea4da640 1 Processor -- start 2024-04-24T23:49:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.123+0000 7fbcea4da640 1 -- start start 2024-04-24T23:49:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.124+0000 7fbcea4da640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbce4144120 0x7fbce406cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.124+0000 7fbcea4da640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbce4072b50 0x7fbce4073000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.124+0000 7fbcea4da640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbce406eda0 con 0x7fbce40cf530 2024-04-24T23:49:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.124+0000 7fbcea4da640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fbce4144500 con 0x7fbce4072b50 2024-04-24T23:49:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.124+0000 7fbcea4da640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fbce40cfcd0 con 0x7fbce4144120 2024-04-24T23:49:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.125+0000 7fbce8a77640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbce4072b50 0x7fbce4073000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.125+0000 7fbce8a77640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbce4072b50 0x7fbce4073000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48560/0 (socket says 172.21.15.16:48560) 2024-04-24T23:49:23.146 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.125+0000 7fbce8a77640 1 -- 172.21.15.16:0/4134411338 learned_addr learned my addr 172.21.15.16:0/4134411338 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.125+0000 7fbce8a77640 1 -- 172.21.15.16:0/4134411338 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbce4144120 msgr2=0x7fbce406cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:49:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.125+0000 7fbce8a77640 1 --2- 172.21.15.16:0/4134411338 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbce4144120 0x7fbce406cf50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.125+0000 7fbce8a77640 1 -- 172.21.15.16:0/4134411338 >> v1:172.21.15.156:6789/0 conn(0x7fbce40cf530 legacy=0x7fbce40b3dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T23:49:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.125+0000 7fbce8a77640 1 -- 172.21.15.16:0/4134411338 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce406f080 con 0x7fbce4072b50 2024-04-24T23:49:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.126+0000 7fbce8a77640 1 --2- 172.21.15.16:0/4134411338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce4072b50 0x7fbce4073000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc00b1a0 tx=0x7fbcdc02a940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07f8bb4895f2608 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.127+0000 7fbce2ffd640 1 -- 172.21.15.16:0/4134411338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc0303f0 con 0x7fbce4072b50 2024-04-24T23:49:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.127+0000 7fbce2ffd640 1 -- 172.21.15.16:0/4134411338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcdc035070 con 0x7fbce4072b50 2024-04-24T23:49:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.128+0000 7fbcea4da640 1 -- 172.21.15.16:0/4134411338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce4072b50 msgr2=0x7fbce4073000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.128+0000 7fbcea4da640 1 --2- 172.21.15.16:0/4134411338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce4072b50 0x7fbce4073000 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc00b1a0 tx=0x7fbcdc02a940 comp rx=0 tx=0).stop 2024-04-24T23:49:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.129+0000 7fbcea4da640 1 -- 172.21.15.16:0/4134411338 shutdown_connections 2024-04-24T23:49:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.129+0000 7fbcea4da640 1 --2- 172.21.15.16:0/4134411338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce4072b50 0x7fbce4073000 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.129+0000 7fbcea4da640 1 --2- 172.21.15.16:0/4134411338 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbce4144120 0x7fbce406cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.129+0000 7fbcea4da640 1 -- 172.21.15.16:0/4134411338 >> 172.21.15.16:0/4134411338 conn(0x7fbce40cd770 msgr2=0x7fbce40cdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.129+0000 7fbcea4da640 1 -- 172.21.15.16:0/4134411338 shutdown_connections 2024-04-24T23:49:23.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.129+0000 7fbcea4da640 1 -- 172.21.15.16:0/4134411338 wait complete. 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.131+0000 7fbcea4da640 1 Processor -- start 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.131+0000 7fbcea4da640 1 -- start start 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.133+0000 7fbcea4da640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce4072b50 0x7fbce40aa520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.133+0000 7fbcea4da640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbce40cf530 0x7fbce413cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.133+0000 7fbcea4da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce4144120 0x7fbce413d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.133+0000 7fbce37fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbce40cf530 0x7fbce413cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.133+0000 7fbce3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce4072b50 0x7fbce40aa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.133+0000 7fbce37fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbce40cf530 0x7fbce413cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:41598/0 (socket says 172.21.15.16:41598) 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.133+0000 7fbce8a77640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce4144120 0x7fbce413d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.134+0000 7fbce8a77640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce4144120 0x7fbce413d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48566/0 (socket says 172.21.15.16:48566) 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.134+0000 7fbce3fff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce4072b50 0x7fbce40aa520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:35004/0 (socket says 172.21.15.16:35004) 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.134+0000 7fbce37fe640 1 -- 172.21.15.16:0/4014661529 learned_addr learned my addr 172.21.15.16:0/4014661529 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.134+0000 7fbcea4da640 1 -- 172.21.15.16:0/4014661529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbce40cf990 con 0x7fbce4144120 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.134+0000 7fbcea4da640 1 -- 172.21.15.16:0/4014661529 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbce40b59b0 con 0x7fbce4072b50 2024-04-24T23:49:23.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.134+0000 7fbcea4da640 1 -- 172.21.15.16:0/4014661529 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fbce40b5e40 con 0x7fbce40cf530 2024-04-24T23:49:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.134+0000 7fbce8a77640 1 -- 172.21.15.16:0/4014661529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbce40cf530 msgr2=0x7fbce413cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.134+0000 7fbce8a77640 1 --2- 172.21.15.16:0/4014661529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbce40cf530 0x7fbce413cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.134+0000 7fbce8a77640 1 -- 172.21.15.16:0/4014661529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce4072b50 msgr2=0x7fbce40aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.134+0000 7fbce8a77640 1 --2- 172.21.15.16:0/4014661529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce4072b50 0x7fbce40aa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.134+0000 7fbce8a77640 1 -- 172.21.15.16:0/4014661529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce40aacc0 con 0x7fbce4144120 2024-04-24T23:49:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.135+0000 7fbce8a77640 1 --2- 172.21.15.16:0/4014661529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce4144120 0x7fbce413d010 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc031000 tx=0x7fbcdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.136+0000 7fbce0ff9640 1 -- 172.21.15.16:0/4014661529 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc004240 con 0x7fbce4144120 2024-04-24T23:49:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.136+0000 7fbcea4da640 1 -- 172.21.15.16:0/4014661529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce40aafa0 con 0x7fbce4144120 2024-04-24T23:49:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.136+0000 7fbcea4da640 1 -- 172.21.15.16:0/4014661529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce40ab560 con 0x7fbce4144120 2024-04-24T23:49:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.139+0000 7fbce0ff9640 1 -- 172.21.15.16:0/4014661529 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbcdc0043e0 con 0x7fbce4144120 2024-04-24T23:49:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.139+0000 7fbce0ff9640 1 -- 172.21.15.16:0/4014661529 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcdc0047d0 con 0x7fbce4144120 2024-04-24T23:49:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.139+0000 7fbce0ff9640 1 -- 172.21.15.16:0/4014661529 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbcdc04c430 con 0x7fbce4144120 2024-04-24T23:49:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.139+0000 7fbce0ff9640 1 --2- 172.21.15.16:0/4014661529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbccc0049b0 0x7fbccc004e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.141+0000 7fbce0ff9640 1 -- 172.21.15.16:0/4014661529 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fbcdc0ac670 con 0x7fbce4144120 2024-04-24T23:49:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.141+0000 7fbce3fff640 1 --2- 172.21.15.16:0/4014661529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbccc0049b0 0x7fbccc004e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.141+0000 7fbce3fff640 1 --2- 172.21.15.16:0/4014661529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbccc0049b0 0x7fbccc004e60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbce413a880 tx=0x7fbcd8009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.141+0000 7fbcea4da640 1 -- 172.21.15.16:0/4014661529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcc400daa0 con 0x7fbce4144120 2024-04-24T23:49:23.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.151+0000 7fbce0ff9640 1 -- 172.21.15.16:0/4014661529 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbcdc07a7c0 con 0x7fbce4144120 2024-04-24T23:49:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.184+0000 7f4077695640 1 Processor -- start 2024-04-24T23:49:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.185+0000 7fe16ffff640 1 Processor -- start 2024-04-24T23:49:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.185+0000 7f4077695640 1 -- start start 2024-04-24T23:49:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.186+0000 7f4077695640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40680ead00 0x7f406804fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.186+0000 7f4077695640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f406804ffd0 0x7f4068037bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.186+0000 7f4077695640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4068038110 0x7f40680385a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.186+0000 7f4077695640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f4068054330 con 0x7f40680ead00 2024-04-24T23:49:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.186+0000 7f4077695640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f4068053720 con 0x7f406804ffd0 2024-04-24T23:49:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.186+0000 7f4077695640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f4068053a00 con 0x7f4068038110 2024-04-24T23:49:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.186+0000 7f28f759e640 1 Processor -- start 2024-04-24T23:49:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.186+0000 7f28f759e640 1 -- start start 2024-04-24T23:49:23.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.185+0000 7fe16ffff640 1 -- start start 2024-04-24T23:49:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7fe16ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe1680ead00 0x7fe16804fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7fe16ffff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe16804ffd0 0x7fe168037bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7fe16ffff640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe168053720 con 0x7fe168038110 2024-04-24T23:49:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7fe16ffff640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fe168054330 con 0x7fe1680ead00 2024-04-24T23:49:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7fe16ffff640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fe168053a00 con 0x7fe16804ffd0 2024-04-24T23:49:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7fe16effd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe1680ead00 0x7fe16804fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7fe16effd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe1680ead00 0x7fe16804fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48578/0 (socket says 172.21.15.16:48578) 2024-04-24T23:49:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7fe16effd640 1 -- 172.21.15.16:0/990405932 learned_addr learned my addr 172.21.15.16:0/990405932 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7fe16f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe168038110 0x7fe168038580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50666/0 (socket says 172.21.15.16:50666) 2024-04-24T23:49:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.188+0000 7fe16e7fc640 1 --2- 172.21.15.16:0/990405932 >> v2:172.21.15.156:3300/0 conn(0x7fe16804ffd0 0x7fe168037bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.188+0000 7fe16effd640 1 -- 172.21.15.16:0/990405932 >> v2:172.21.15.156:3300/0 conn(0x7fe16804ffd0 msgr2=0x7fe168037bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.188+0000 7fe16effd640 1 --2- 172.21.15.16:0/990405932 >> v2:172.21.15.156:3300/0 conn(0x7fe16804ffd0 0x7fe168037bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.188+0000 7fe16effd640 1 -- 172.21.15.16:0/990405932 >> v1:172.21.15.16:6789/0 conn(0x7fe168038110 legacy=0x7fe168038580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.188+0000 7fe16effd640 1 -- 172.21.15.16:0/990405932 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe168053ce0 con 0x7fe1680ead00 2024-04-24T23:49:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7fb8fc8b9640 1 Processor -- start 2024-04-24T23:49:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.188+0000 7fb8fc8b9640 1 -- start start 2024-04-24T23:49:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7f4075431640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40680ead00 0x7f406804fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7f4075431640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40680ead00 0x7f406804fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48574/0 (socket says 172.21.15.16:48574) 2024-04-24T23:49:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7f4075431640 1 -- 172.21.15.16:0/2931859847 learned_addr learned my addr 172.21.15.16:0/2931859847 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7f4075c32640 1 --2- 172.21.15.16:0/2931859847 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4068038110 0x7f40680385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7f4074c30640 1 --2- 172.21.15.16:0/2931859847 >> v2:172.21.15.156:3300/0 conn(0x7f406804ffd0 0x7f4068037bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7f4075c32640 1 -- 172.21.15.16:0/2931859847 >> v2:172.21.15.156:3300/0 conn(0x7f406804ffd0 msgr2=0x7f4068037bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7f4075c32640 1 --2- 172.21.15.16:0/2931859847 >> v2:172.21.15.156:3300/0 conn(0x7f406804ffd0 0x7f4068037bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7f4075c32640 1 -- 172.21.15.16:0/2931859847 >> v2:172.21.15.16:3300/0 conn(0x7f40680ead00 msgr2=0x7f406804fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.188+0000 7f4075c32640 1 --2- 172.21.15.16:0/2931859847 >> v2:172.21.15.16:3300/0 conn(0x7f40680ead00 0x7f406804fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.188+0000 7f4075c32640 1 -- 172.21.15.16:0/2931859847 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4068053ce0 con 0x7f4068038110 2024-04-24T23:49:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.188+0000 7f4074c30640 1 --2- 172.21.15.16:0/2931859847 >> v2:172.21.15.156:3300/0 conn(0x7f406804ffd0 0x7f4068037bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7fb38359e640 1 Processor -- start 2024-04-24T23:49:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.187+0000 7fb38359e640 1 -- start start 2024-04-24T23:49:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7f57e0d63640 1 Processor -- start 2024-04-24T23:49:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7f57e0d63640 1 -- start start 2024-04-24T23:49:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7f28f759e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f28f00e3260 0x7f28f004f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb8fc8b9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f00069e0 0x7fb8f0006e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb8fc8b9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8f0053720 con 0x7fb8f00ead00 2024-04-24T23:49:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb8fc8b9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8f0053a00 con 0x7fb8f0005e90 2024-04-24T23:49:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb8fc8b9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb8f0054330 con 0x7fb8f00069e0 2024-04-24T23:49:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7f4075c32640 1 --2- 172.21.15.16:0/2931859847 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4068038110 0x7f40680385a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4070000b60 tx=0x7f40700a5a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e45fb4e8fd4728b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb38359e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb37c0ead00 0x7fb37c04fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb38359e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb37c04ffd0 0x7fb37c037bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb38359e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb37c053720 con 0x7fb37c038110 2024-04-24T23:49:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb38359e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb37c054330 con 0x7fb37c04ffd0 2024-04-24T23:49:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb38359e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb37c053a00 con 0x7fb37c0ead00 2024-04-24T23:49:23.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb8f759e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f00069e0 0x7fb8f0006e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb8f759e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f00069e0 0x7fb8f0006e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:41618/0 (socket says 172.21.15.16:41618) 2024-04-24T23:49:23.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb8f759e640 1 -- 172.21.15.16:0/2181329619 learned_addr learned my addr 172.21.15.16:0/2181329619 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7f4067fff640 1 -- 172.21.15.16:0/2931859847 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40700a9900 con 0x7f4068038110 2024-04-24T23:49:23.194 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7f28f759e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28f00e3640 con 0x7f28f0053380 2024-04-24T23:49:23.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb381d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb37c04ffd0 0x7fb37c037bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb381d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb37c04ffd0 0x7fb37c037bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48584/0 (socket says 172.21.15.16:48584) 2024-04-24T23:49:23.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7fb381d9b640 1 -- 172.21.15.16:0/3932346649 learned_addr learned my addr 172.21.15.16:0/3932346649 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.189+0000 7f28f659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f28f00e3260 0x7f28f004f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7f28f5d9b640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f28f004fdb0 0x7f28f0052c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:41352/0 (socket says 172.21.15.16:41352) 2024-04-24T23:49:23.195 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7f28f5d9b640 1 -- 172.21.15.16:0/4016893106 learned_addr learned my addr 172.21.15.16:0/4016893106 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:23.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7fb38259c640 1 --2- 172.21.15.16:0/3932346649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb37c0ead00 0x7fb37c04fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7f4067fff640 1 -- 172.21.15.16:0/2931859847 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40700a9ae0 con 0x7f4068038110 2024-04-24T23:49:23.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7f28f759e640 1 -- 172.21.15.16:0/4016893106 --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28f0053930 con 0x7f28f004fdb0 2024-04-24T23:49:23.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7fe16effd640 1 --2- 172.21.15.16:0/990405932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1680ead00 0x7fe16804fa90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe168036560 tx=0x7fe16402a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=865086f9085ac44b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7f28f759e640 1 -- 172.21.15.16:0/4016893106 --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f28f0053c10 con 0x7f28f00e3260 2024-04-24T23:49:23.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7fb38259c640 1 -- 172.21.15.16:0/3932346649 >> v2:172.21.15.16:3300/0 conn(0x7fb37c04ffd0 msgr2=0x7fb37c037bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:23.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7fb38259c640 1 --2- 172.21.15.16:0/3932346649 >> v2:172.21.15.16:3300/0 conn(0x7fb37c04ffd0 0x7fb37c037bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7fb38259c640 1 -- 172.21.15.16:0/3932346649 >> v1:172.21.15.156:6789/0 conn(0x7fb37c038110 legacy=0x7fb37c0385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7fb38259c640 1 -- 172.21.15.16:0/3932346649 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37c053ce0 con 0x7fb37c0ead00 2024-04-24T23:49:23.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7f28f659c640 1 -- 172.21.15.16:0/4016893106 >> v1:172.21.15.156:6789/0 conn(0x7f28f004fdb0 legacy=0x7f28f0052c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7f28f659c640 1 -- 172.21.15.16:0/4016893106 >> v1:172.21.15.16:6789/0 conn(0x7f28f0053380 legacy=0x7f28f0037bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:23.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f28f659c640 1 -- 172.21.15.16:0/4016893106 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f0053ef0 con 0x7f28f00e3260 2024-04-24T23:49:23.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f4077695640 1 -- 172.21.15.16:0/2931859847 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4068038110 msgr2=0x7f40680385a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f4077695640 1 --2- 172.21.15.16:0/2931859847 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4068038110 0x7f40680385a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4070000b60 tx=0x7f40700a5a80 comp rx=0 tx=0).stop 2024-04-24T23:49:23.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7fe16dffb640 1 -- 172.21.15.16:0/990405932 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164037070 con 0x7fe1680ead00 2024-04-24T23:49:23.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7fe16dffb640 1 -- 172.21.15.16:0/990405932 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16402adf0 con 0x7fe1680ead00 2024-04-24T23:49:23.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7fe16dffb640 1 -- 172.21.15.16:0/990405932 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe164032660 con 0x7fe1680ead00 2024-04-24T23:49:23.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7f57e0d63640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57dc15c490 0x7f57dc0b5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.190+0000 7f57e0d63640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57dc0cf220 0x7f57dc0cf690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f57e0d63640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57dc160540 con 0x7f57dc15c0b0 2024-04-24T23:49:23.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f57e0d63640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f57dc0cfbd0 con 0x7f57dc15c490 2024-04-24T23:49:23.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f57e0d63640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f57dc160820 con 0x7f57dc0cf220 2024-04-24T23:49:23.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f57d9d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57dc15c490 0x7f57dc0b5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f57d9d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57dc15c490 0x7f57dc0b5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48602/0 (socket says 172.21.15.16:48602) 2024-04-24T23:49:23.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f57d9d9b640 1 -- 172.21.15.16:0/2892635475 learned_addr learned my addr 172.21.15.16:0/2892635475 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7fb8f5d9b640 1 -- 172.21.15.16:0/2181329619 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4074207894 0 0) 0x7fb8f0053a00 con 0x7fb8f0005e90 2024-04-24T23:49:23.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f4077695640 1 -- 172.21.15.16:0/2931859847 shutdown_connections 2024-04-24T23:49:23.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f4077695640 1 --2- 172.21.15.16:0/2931859847 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4068038110 0x7f40680385a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f4077695640 1 --2- 172.21.15.16:0/2931859847 >> v2:172.21.15.156:3300/0 conn(0x7f406804ffd0 0x7f4068037bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f4077695640 1 --2- 172.21.15.16:0/2931859847 >> v2:172.21.15.16:3300/0 conn(0x7f40680ead00 0x7f406804fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f4077695640 1 -- 172.21.15.16:0/2931859847 >> 172.21.15.16:0/2931859847 conn(0x7f4068009fb0 msgr2=0x7f406800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7fb8f5d9b640 1 -- 172.21.15.16:0/2181329619 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8e0003520 con 0x7fb8f0005e90 2024-04-24T23:49:23.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7fb38259c640 1 --2- 172.21.15.16:0/3932346649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb37c0ead00 0x7fb37c04fa90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb384050770 tx=0x7fb3840a5a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e892b406cc8b1e13 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f4077695640 1 -- 172.21.15.16:0/2931859847 shutdown_connections 2024-04-24T23:49:23.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7fb38159a640 1 -- 172.21.15.16:0/3932346649 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3840a98e0 con 0x7fb37c0ead00 2024-04-24T23:49:23.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fb8f5d9b640 1 -- 172.21.15.16:0/2181329619 <== mon.1 v1:172.21.15.156:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 391153980 0 0) 0x7fb8e0003520 con 0x7fb8f0005e90 2024-04-24T23:49:23.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fb38159a640 1 -- 172.21.15.16:0/3932346649 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3840a9ac0 con 0x7fb37c0ead00 2024-04-24T23:49:23.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fb38159a640 1 -- 172.21.15.16:0/3932346649 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3840b1400 con 0x7fb37c0ead00 2024-04-24T23:49:23.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fb8f5d9b640 1 -- 172.21.15.16:0/2181329619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f00069e0 msgr2=0x7fb8f0006e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:23.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fb8f5d9b640 1 --2- 172.21.15.16:0/2181329619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f00069e0 0x7fb8f0006e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fb8f5d9b640 1 -- 172.21.15.16:0/2181329619 >> v1:172.21.15.16:6789/0 conn(0x7fb8f00ead00 legacy=0x7fb8f004fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fe16ffff640 1 -- 172.21.15.16:0/990405932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1680ead00 msgr2=0x7fe16804fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fe16ffff640 1 --2- 172.21.15.16:0/990405932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1680ead00 0x7fe16804fa90 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe168036560 tx=0x7fe16402a160 comp rx=0 tx=0).stop 2024-04-24T23:49:23.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7f28f659c640 1 --2- 172.21.15.16:0/4016893106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f00e3260 0x7f28f004f870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f28e80097d0 tx=0x7f28e802a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa97dfeb077eb71b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fb8f5d9b640 1 -- 172.21.15.16:0/2181329619 --> v1:172.21.15.156:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8f0053ce0 con 0x7fb8f0005e90 2024-04-24T23:49:23.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fb8f5d9b640 1 -- 172.21.15.16:0/2181329619 <== mon.1 v1:172.21.15.156:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7fb8e8001670 con 0x7fb8f0005e90 2024-04-24T23:49:23.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fe16ffff640 1 -- 172.21.15.16:0/990405932 shutdown_connections 2024-04-24T23:49:23.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fe16ffff640 1 --2- 172.21.15.16:0/990405932 >> v2:172.21.15.156:3300/0 conn(0x7fe16804ffd0 0x7fe168037bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fe16ffff640 1 --2- 172.21.15.16:0/990405932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe1680ead00 0x7fe16804fa90 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fe16ffff640 1 -- 172.21.15.16:0/990405932 >> 172.21.15.16:0/990405932 conn(0x7fe168009fb0 msgr2=0x7fe16800a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fb8f5d9b640 1 -- 172.21.15.16:0/2181329619 >> v1:172.21.15.156:6789/0 conn(0x7fb8f0005e90 legacy=0x7fb8f00062d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.191+0000 7f4077695640 1 -- 172.21.15.16:0/2931859847 wait complete. 2024-04-24T23:49:23.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fb8f5d9b640 1 --2- 172.21.15.16:0/2181329619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e00039b0 0x7fb8e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7f28f559a640 1 -- 172.21.15.16:0/4016893106 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8037070 con 0x7f28f00e3260 2024-04-24T23:49:23.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7fb8f5d9b640 1 --2- 172.21.15.16:0/2181329619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8e00043b0 0x7fb8e0004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7fb8f5d9b640 1 --2- 172.21.15.16:0/2181329619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f00069e0 0x7fb8e00050e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7fb8f5d9b640 1 -- 172.21.15.16:0/2181329619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f0054330 con 0x7fb8e00039b0 2024-04-24T23:49:23.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7f780e407640 1 Processor -- start 2024-04-24T23:49:23.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7f780e407640 1 -- start start 2024-04-24T23:49:23.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7f57d9d9b640 1 -- 172.21.15.16:0/2892635475 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57dc0cf220 msgr2=0x7f57dc0cf690 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T23:49:23.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7f57d9d9b640 1 --2- 172.21.15.16:0/2892635475 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57dc0cf220 0x7f57dc0cf690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7f57d9d9b640 1 -- 172.21.15.16:0/2892635475 >> v1:172.21.15.16:6789/0 conn(0x7f57dc15c0b0 legacy=0x7f57dc15fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.192+0000 7f57d9d9b640 1 -- 172.21.15.16:0/2892635475 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57dc0b4810 con 0x7f57dc15c490 2024-04-24T23:49:23.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7fb8f5d9b640 1 -- 172.21.15.16:0/2181329619 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb8e000b9f0 con 0x7fb8e00043b0 2024-04-24T23:49:23.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7fb8f5d9b640 1 -- 172.21.15.16:0/2181329619 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb8e000bbf0 con 0x7fb8f00069e0 2024-04-24T23:49:23.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7f28f559a640 1 -- 172.21.15.16:0/4016893106 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e8032440 con 0x7f28f00e3260 2024-04-24T23:49:23.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7f28f559a640 1 -- 172.21.15.16:0/4016893106 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8032810 con 0x7f28f00e3260 2024-04-24T23:49:23.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7f57d9d9b640 1 --2- 172.21.15.16:0/2892635475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57dc15c490 0x7f57dc0b5010 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f57cc009870 tx=0x7f57cc02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cd12d41fea192f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7f4077695640 1 Processor -- start 2024-04-24T23:49:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7fb8f659c640 1 --2- 172.21.15.16:0/2181329619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8e00043b0 0x7fb8e0004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7fb8f759e640 1 --2- 172.21.15.16:0/2181329619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f00069e0 0x7fb8e00050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7fb8f6d9d640 1 --2- 172.21.15.16:0/2181329619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e00039b0 0x7fb8e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7fb38359e640 1 -- 172.21.15.16:0/3932346649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb37c0ead00 msgr2=0x7fb37c04fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7fb38359e640 1 --2- 172.21.15.16:0/3932346649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb37c0ead00 0x7fb37c04fa90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb384050770 tx=0x7fb3840a5a60 comp rx=0 tx=0).stop 2024-04-24T23:49:23.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7fe16ffff640 1 -- 172.21.15.16:0/990405932 shutdown_connections 2024-04-24T23:49:23.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7fb8f759e640 1 -- 172.21.15.16:0/2181329619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8e00043b0 msgr2=0x7fb8e0004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:23.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7fb8f759e640 1 --2- 172.21.15.16:0/2181329619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8e00043b0 0x7fb8e0004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7fb8f759e640 1 -- 172.21.15.16:0/2181329619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e00039b0 msgr2=0x7fb8e0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:23.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7fb8f759e640 1 --2- 172.21.15.16:0/2181329619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e00039b0 0x7fb8e0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7fb8f759e640 1 -- 172.21.15.16:0/2181329619 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8e0003520 con 0x7fb8f00069e0 2024-04-24T23:49:23.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7fb8f6d9d640 1 --2- 172.21.15.16:0/2181329619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8e00039b0 0x7fb8e0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:23.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7fb38359e640 1 -- 172.21.15.16:0/3932346649 shutdown_connections 2024-04-24T23:49:23.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7fb38359e640 1 --2- 172.21.15.16:0/3932346649 >> v2:172.21.15.16:3300/0 conn(0x7fb37c04ffd0 0x7fb37c037bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7fb38359e640 1 --2- 172.21.15.16:0/3932346649 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb37c0ead00 0x7fb37c04fa90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7fb38359e640 1 -- 172.21.15.16:0/3932346649 >> 172.21.15.16:0/3932346649 conn(0x7fb37c009fb0 msgr2=0x7fb37c00a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f780e407640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78080ce080 0x7f7808163a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f57d959a640 1 -- 172.21.15.16:0/2892635475 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc037070 con 0x7f57dc15c490 2024-04-24T23:49:23.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f57d959a640 1 -- 172.21.15.16:0/2892635475 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57cc032440 con 0x7f57dc15c490 2024-04-24T23:49:23.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f57d959a640 1 -- 172.21.15.16:0/2892635475 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc032810 con 0x7f57dc15c490 2024-04-24T23:49:23.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f28f759e640 1 -- 172.21.15.16:0/4016893106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f00e3260 msgr2=0x7f28f004f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f28f759e640 1 --2- 172.21.15.16:0/4016893106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f00e3260 0x7f28f004f870 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f28e80097d0 tx=0x7f28e802a2a0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f7807fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78080ce080 0x7f7808163a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f7807fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78080ce080 0x7f7808163a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48614/0 (socket says 172.21.15.16:48614) 2024-04-24T23:49:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f780e407640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7808162c80 con 0x7f78080b3cc0 2024-04-24T23:49:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f780e407640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7808162f60 con 0x7f78080ce460 2024-04-24T23:49:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f780e407640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f7808163240 con 0x7f78080ce080 2024-04-24T23:49:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7fb38359e640 1 -- 172.21.15.16:0/3932346649 shutdown_connections 2024-04-24T23:49:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7fb38359e640 1 -- 172.21.15.16:0/3932346649 wait complete. 2024-04-24T23:49:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f28f759e640 1 -- 172.21.15.16:0/4016893106 shutdown_connections 2024-04-24T23:49:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f28f759e640 1 --2- 172.21.15.16:0/4016893106 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f00e3260 0x7f28f004f870 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f28f759e640 1 -- 172.21.15.16:0/4016893106 >> 172.21.15.16:0/4016893106 conn(0x7f28f0009fb0 msgr2=0x7f28f000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.194+0000 7f7807fff640 1 -- 172.21.15.16:0/3605633947 learned_addr learned my addr 172.21.15.16:0/3605633947 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7f4077695640 1 -- start start 2024-04-24T23:49:23.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f4077695640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4068038110 0x7f40680365b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f4077695640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f406804ffd0 0x7f406804ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f4077695640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40680ead00 0x7f4068188f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f7807fff640 1 -- 172.21.15.16:0/3605633947 >> v1:172.21.15.156:6789/0 conn(0x7f78080ce460 legacy=0x7f7808071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f7807fff640 1 -- 172.21.15.16:0/3605633947 >> v1:172.21.15.16:6789/0 conn(0x7f78080b3cc0 legacy=0x7f78080b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:23.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f7807fff640 1 -- 172.21.15.16:0/3605633947 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7808163520 con 0x7f78080ce080 2024-04-24T23:49:23.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f4077695640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4068054940 con 0x7f40680ead00 2024-04-24T23:49:23.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f4077695640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f40680547c0 con 0x7f406804ffd0 2024-04-24T23:49:23.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f4077695640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f406804f550 con 0x7f4068038110 2024-04-24T23:49:23.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f57e0d63640 1 -- 172.21.15.16:0/2892635475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57dc15c490 msgr2=0x7f57dc0b5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f57e0d63640 1 --2- 172.21.15.16:0/2892635475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57dc15c490 0x7f57dc0b5010 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f57cc009870 tx=0x7f57cc02a270 comp rx=0 tx=0).stop 2024-04-24T23:49:23.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.193+0000 7fe16ffff640 1 -- 172.21.15.16:0/990405932 wait complete. 2024-04-24T23:49:23.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f4075431640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4068038110 0x7f40680365b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f4075431640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4068038110 0x7f40680365b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:41656/0 (socket says 172.21.15.16:41656) 2024-04-24T23:49:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f4075431640 1 -- 172.21.15.16:0/2220237329 learned_addr learned my addr 172.21.15.16:0/2220237329 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f4074c30640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f406804ffd0 0x7f406804ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.195+0000 7f4075c32640 1 --2- 172.21.15.16:0/2220237329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40680ead00 0x7f4068188f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7fe16ffff640 1 Processor -- start 2024-04-24T23:49:23.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f7807fff640 1 --2- 172.21.15.16:0/3605633947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080ce080 0x7f7808163a60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7800009e50 tx=0x7f780002a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8299474a2ffe4fb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f4075c32640 1 -- 172.21.15.16:0/2220237329 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4068038110 msgr2=0x7f40680365b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f4075c32640 1 --2- 172.21.15.16:0/2220237329 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4068038110 0x7f40680365b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f4075c32640 1 -- 172.21.15.16:0/2220237329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f406804ffd0 msgr2=0x7f406804ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f4075c32640 1 --2- 172.21.15.16:0/2220237329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f406804ffd0 0x7f406804ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f4075c32640 1 -- 172.21.15.16:0/2220237329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40680dd670 con 0x7f40680ead00 2024-04-24T23:49:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f57e0d63640 1 -- 172.21.15.16:0/2892635475 shutdown_connections 2024-04-24T23:49:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f57e0d63640 1 --2- 172.21.15.16:0/2892635475 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57dc0cf220 0x7f57dc0cf690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f57e0d63640 1 --2- 172.21.15.16:0/2892635475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57dc15c490 0x7f57dc0b5010 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f57e0d63640 1 -- 172.21.15.16:0/2892635475 >> 172.21.15.16:0/2892635475 conn(0x7f57dc0cdcc0 msgr2=0x7f57dc06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f7806ffd640 1 -- 172.21.15.16:0/3605633947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7800036070 con 0x7f78080ce080 2024-04-24T23:49:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f7806ffd640 1 -- 172.21.15.16:0/3605633947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7800031600 con 0x7f78080ce080 2024-04-24T23:49:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f7806ffd640 1 -- 172.21.15.16:0/3605633947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78000319d0 con 0x7f78080ce080 2024-04-24T23:49:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7fb38359e640 1 Processor -- start 2024-04-24T23:49:23.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f28f759e640 1 -- 172.21.15.16:0/4016893106 shutdown_connections 2024-04-24T23:49:23.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7fb8f759e640 1 --2- 172.21.15.16:0/2181329619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f00069e0 0x7fb8e00050e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb8f80502b0 tx=0x7fb8e0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de2927514707866e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f28f759e640 1 -- 172.21.15.16:0/4016893106 wait complete. 2024-04-24T23:49:23.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f57e0d63640 1 -- 172.21.15.16:0/2892635475 shutdown_connections 2024-04-24T23:49:23.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f57e0d63640 1 -- 172.21.15.16:0/2892635475 wait complete. 2024-04-24T23:49:23.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7fe16ffff640 1 -- start start 2024-04-24T23:49:23.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7f4075c32640 1 --2- 172.21.15.16:0/2220237329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40680ead00 0x7f4068188f50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f407007a980 tx=0x7f407004fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.197+0000 7f4065ffb640 1 -- 172.21.15.16:0/2220237329 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40700b1c20 con 0x7f40680ead00 2024-04-24T23:49:23.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.197+0000 7f4065ffb640 1 -- 172.21.15.16:0/2220237329 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40700b1dc0 con 0x7f40680ead00 2024-04-24T23:49:23.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.197+0000 7f4065ffb640 1 -- 172.21.15.16:0/2220237329 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40700bc610 con 0x7f40680ead00 2024-04-24T23:49:23.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.197+0000 7f780e407640 1 -- 172.21.15.16:0/3605633947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080ce080 msgr2=0x7f7808163a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.197+0000 7f780e407640 1 --2- 172.21.15.16:0/3605633947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080ce080 0x7f7808163a60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7800009e50 tx=0x7f780002a6d0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.197+0000 7f4077695640 1 -- 172.21.15.16:0/2220237329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40680dd950 con 0x7f40680ead00 2024-04-24T23:49:23.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.197+0000 7f780e407640 1 -- 172.21.15.16:0/3605633947 shutdown_connections 2024-04-24T23:49:23.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.197+0000 7f780e407640 1 --2- 172.21.15.16:0/3605633947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080ce080 0x7f7808163a60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.197+0000 7f780e407640 1 -- 172.21.15.16:0/3605633947 >> 172.21.15.16:0/3605633947 conn(0x7f78080b2000 msgr2=0x7f78080cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.197+0000 7fe16ffff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe168038110 0x7fe16802e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.197+0000 7fe16ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16804ffd0 0x7fe16802b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7f780e407640 1 -- 172.21.15.16:0/3605633947 shutdown_connections 2024-04-24T23:49:23.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.196+0000 7fb38359e640 1 -- start start 2024-04-24T23:49:23.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fe16ffff640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1680ead00 0x7fe1680e9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fe16e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16804ffd0 0x7fe16802b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fe16e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16804ffd0 0x7fe16802b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48646/0 (socket says 172.21.15.16:48646) 2024-04-24T23:49:23.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fe16e7fc640 1 -- 172.21.15.16:0/88204952 learned_addr learned my addr 172.21.15.16:0/88204952 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7f28f759e640 1 Processor -- start 2024-04-24T23:49:23.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7f28f759e640 1 -- start start 2024-04-24T23:49:23.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fb8f5d9b640 1 -- 172.21.15.16:0/2181329619 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f80b3070 con 0x7fb8f00069e0 2024-04-24T23:49:23.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fb8f5d9b640 1 -- 172.21.15.16:0/2181329619 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8f80a6ab0 con 0x7fb8f00069e0 2024-04-24T23:49:23.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7f57e0d63640 1 Processor -- start 2024-04-24T23:49:23.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7f57e0d63640 1 -- start start 2024-04-24T23:49:23.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/2181329619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f00069e0 msgr2=0x7fb8e00050e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fb8fc8b9640 1 --2- 172.21.15.16:0/2181329619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f00069e0 0x7fb8e00050e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb8f80502b0 tx=0x7fb8e0005830 comp rx=0 tx=0).stop 2024-04-24T23:49:23.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fb8f759e640 1 -- 172.21.15.16:0/2181329619 reap_dead start 2024-04-24T23:49:23.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fb38359e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb37c038110 0x7fb37c0dcb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fb38359e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb37c04ffd0 0x7fb37c04e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fb38359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c0ead00 0x7fb37c045a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fb38359e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c0547c0 con 0x7fb37c0ead00 2024-04-24T23:49:23.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/2181329619 shutdown_connections 2024-04-24T23:49:23.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/2181329619 >> 172.21.15.16:0/2181329619 conn(0x7fb8f0009fb0 msgr2=0x7fb8f000a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7f780e407640 1 -- 172.21.15.16:0/3605633947 wait complete. 2024-04-24T23:49:23.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f28f759e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28f004fdb0 0x7f28f00cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f780e407640 1 Processor -- start 2024-04-24T23:49:23.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f28f759e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f0053380 0x7f28f00e3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fb38359e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c038cb0 con 0x7fb37c038110 2024-04-24T23:49:23.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fb38359e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb37c054940 con 0x7fb37c04ffd0 2024-04-24T23:49:23.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fb382d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c0ead00 0x7fb37c045a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fb38259c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb37c038110 0x7fb37c0dcb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fb382d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c0ead00 0x7fb37c045a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48650/0 (socket says 172.21.15.16:48650) 2024-04-24T23:49:23.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fb382d9d640 1 -- 172.21.15.16:0/1455527805 learned_addr learned my addr 172.21.15.16:0/1455527805 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fb381d9b640 1 --2- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb37c04ffd0 0x7fb37c04e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fb381d9b640 1 -- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb37c038110 msgr2=0x7fb37c0dcb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fb381d9b640 1 --2- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb37c038110 0x7fb37c0dcb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fb381d9b640 1 -- 172.21.15.16:0/1455527805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c0ead00 msgr2=0x7fb37c045a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fb381d9b640 1 --2- 172.21.15.16:0/1455527805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c0ead00 0x7fb37c045a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fb381d9b640 1 -- 172.21.15.16:0/1455527805 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37c0e8a40 con 0x7fb37c04ffd0 2024-04-24T23:49:23.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fb38259c640 1 --2- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb37c038110 0x7fb37c0dcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:23.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7fb381d9b640 1 --2- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb37c04ffd0 0x7fb37c04e550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb36c00e600 tx=0x7fb36c00eac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fe16ffff640 1 -- 172.21.15.16:0/88204952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe1680547c0 con 0x7fe16804ffd0 2024-04-24T23:49:23.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fe16ffff640 1 -- 172.21.15.16:0/88204952 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe168038c90 con 0x7fe1680ead00 2024-04-24T23:49:23.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fe16ffff640 1 -- 172.21.15.16:0/88204952 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe168054940 con 0x7fe168038110 2024-04-24T23:49:23.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fe16f7fe640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1680ead00 0x7fe1680e9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.198+0000 7fe16effd640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe168038110 0x7fe16802e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fe16e7fc640 1 -- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe168038110 msgr2=0x7fe16802e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fe16e7fc640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe168038110 0x7fe16802e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fe16e7fc640 1 -- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1680ead00 msgr2=0x7fe1680e9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fe16e7fc640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1680ead00 0x7fe1680e9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fe16e7fc640 1 -- 172.21.15.16:0/88204952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe168189020 con 0x7fe16804ffd0 2024-04-24T23:49:23.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fe16f7fe640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1680ead00 0x7fe1680e9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:23.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.199+0000 7fe16effd640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe168038110 0x7fe16802e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:23.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f28f759e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28f00e3260 0x7f28f00e3d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f28f759e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f28f00382c0 con 0x7f28f0053380 2024-04-24T23:49:23.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f28f759e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f28f00544b0 con 0x7f28f004fdb0 2024-04-24T23:49:23.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f28f759e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f28f0038440 con 0x7f28f00e3260 2024-04-24T23:49:23.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f28f659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28f004fdb0 0x7f28f00cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f28f659c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28f004fdb0 0x7f28f00cdef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:35054/0 (socket says 172.21.15.16:35054) 2024-04-24T23:49:23.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f28f659c640 1 -- 172.21.15.16:0/436300424 learned_addr learned my addr 172.21.15.16:0/436300424 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f57e0d63640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57dc0cf220 0x7f57dc14bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f780e407640 1 -- start start 2024-04-24T23:49:23.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f28f5d9b640 1 --2- 172.21.15.16:0/436300424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f0053380 0x7f28f00e3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f4077695640 1 -- 172.21.15.16:0/2220237329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40680ddf10 con 0x7f40680ead00 2024-04-24T23:49:23.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/2181329619 shutdown_connections 2024-04-24T23:49:23.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f57e0d63640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57dc15c0b0 0x7f57dc150360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f4065ffb640 1 -- 172.21.15.16:0/2220237329 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f40700c3020 con 0x7f40680ead00 2024-04-24T23:49:23.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f28f659c640 1 -- 172.21.15.16:0/436300424 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28f00e3260 msgr2=0x7f28f00e3d80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:49:23.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f28f659c640 1 --2- 172.21.15.16:0/436300424 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28f00e3260 0x7f28f00e3d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f28f659c640 1 -- 172.21.15.16:0/436300424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f0053380 msgr2=0x7f28f00e3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f28f659c640 1 --2- 172.21.15.16:0/436300424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f0053380 0x7f28f00e3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f28f659c640 1 -- 172.21.15.16:0/436300424 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28f004aff0 con 0x7f28f004fdb0 2024-04-24T23:49:23.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f780e407640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080b3cc0 0x7f780812ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f4065ffb640 1 --2- 172.21.15.16:0/2220237329 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f405406be80 0x7f405406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7fe16e7fc640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16804ffd0 0x7fe16802b960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe16000e9e0 tx=0x7fe16000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f4065ffb640 1 -- 172.21.15.16:0/2220237329 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f407012c060 con 0x7f40680ead00 2024-04-24T23:49:23.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7fe1577fe640 1 -- 172.21.15.16:0/88204952 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe160017070 con 0x7fe16804ffd0 2024-04-24T23:49:23.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7fe1577fe640 1 -- 172.21.15.16:0/88204952 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16000c9d0 con 0x7fe16804ffd0 2024-04-24T23:49:23.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7fe1577fe640 1 -- 172.21.15.16:0/88204952 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16000cdc0 con 0x7fe16804ffd0 2024-04-24T23:49:23.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7fe16ffff640 1 -- 172.21.15.16:0/88204952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe168189300 con 0x7fe16804ffd0 2024-04-24T23:49:23.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7f7807fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080b3cc0 0x7f780812ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7f7807fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080b3cc0 0x7f780812ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48676/0 (socket says 172.21.15.16:48676) 2024-04-24T23:49:23.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7f7807fff640 1 -- 172.21.15.16:0/2567191678 learned_addr learned my addr 172.21.15.16:0/2567191678 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7fe16ffff640 1 -- 172.21.15.16:0/88204952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1681898c0 con 0x7fe16804ffd0 2024-04-24T23:49:23.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7f780e407640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78080ce080 0x7f7808200ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7f780e407640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78080ce460 0x7f780806da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7f78077fe640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78080ce080 0x7f7808200ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7f780e407640 1 -- 172.21.15.16:0/2567191678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7808162a20 con 0x7f78080b3cc0 2024-04-24T23:49:23.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7f780e407640 1 -- 172.21.15.16:0/2567191678 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f78081628a0 con 0x7f78080ce460 2024-04-24T23:49:23.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7f780e407640 1 -- 172.21.15.16:0/2567191678 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f78080b4150 con 0x7f78080ce080 2024-04-24T23:49:23.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7f4075431640 1 --2- 172.21.15.16:0/2220237329 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f405406be80 0x7f405406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/2181329619 wait complete. 2024-04-24T23:49:23.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.203+0000 7f28f659c640 1 --2- 172.21.15.16:0/436300424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28f004fdb0 0x7f28f00cdef0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f28e8000c00 tx=0x7f28e80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7fb37affd640 1 -- 172.21.15.16:0/1455527805 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c017070 con 0x7fb37c04ffd0 2024-04-24T23:49:23.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7fb37affd640 1 -- 172.21.15.16:0/1455527805 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb36c00c620 con 0x7fb37c04ffd0 2024-04-24T23:49:23.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7fb37affd640 1 -- 172.21.15.16:0/1455527805 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c00f040 con 0x7fb37c04ffd0 2024-04-24T23:49:23.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f57e0d63640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57dc15c490 0x7f57dc14d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.203+0000 7fb38359e640 1 -- 172.21.15.16:0/1455527805 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb37c043960 con 0x7fb37c04ffd0 2024-04-24T23:49:23.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.203+0000 7fb38359e640 1 -- 172.21.15.16:0/1455527805 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb37c043f20 con 0x7fb37c04ffd0 2024-04-24T23:49:23.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.203+0000 7fb8fc8b9640 1 Processor -- start 2024-04-24T23:49:23.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f57e0d63640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc15c900 con 0x7f57dc15c0b0 2024-04-24T23:49:23.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.203+0000 7f780c9a4640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78080ce460 0x7f780806da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f57e0d63640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f57dc160b00 con 0x7f57dc15c490 2024-04-24T23:49:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f57e0d63640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f57dc15ca80 con 0x7f57dc0cf220 2024-04-24T23:49:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f57da59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57dc0cf220 0x7f57dc14bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f57da59c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57dc0cf220 0x7f57dc14bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:41702/0 (socket says 172.21.15.16:41702) 2024-04-24T23:49:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f57da59c640 1 -- 172.21.15.16:0/3894122336 learned_addr learned my addr 172.21.15.16:0/3894122336 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.200+0000 7f57dad9d640 1 --2- 172.21.15.16:0/3894122336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57dc15c490 0x7f57dc14d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f57d9d9b640 1 --2- 172.21.15.16:0/3894122336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57dc15c0b0 0x7f57dc150360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f57d9d9b640 1 -- 172.21.15.16:0/3894122336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57dc0cf220 msgr2=0x7f57dc14bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f57d9d9b640 1 --2- 172.21.15.16:0/3894122336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57dc0cf220 0x7f57dc14bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f57d9d9b640 1 -- 172.21.15.16:0/3894122336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57dc15c490 msgr2=0x7f57dc14d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f57d9d9b640 1 --2- 172.21.15.16:0/3894122336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57dc15c490 0x7f57dc14d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.201+0000 7f57d9d9b640 1 -- 172.21.15.16:0/3894122336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57dc14a430 con 0x7f57dc15c0b0 2024-04-24T23:49:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7f57d9d9b640 1 --2- 172.21.15.16:0/3894122336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57dc15c0b0 0x7f57dc150360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f57cc031000 tx=0x7f57cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7f57caffd640 1 -- 172.21.15.16:0/3894122336 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc042020 con 0x7f57dc15c0b0 2024-04-24T23:49:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.202+0000 7f57e0d63640 1 -- 172.21.15.16:0/3894122336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57dc14a710 con 0x7f57dc15c0b0 2024-04-24T23:49:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.203+0000 7f57e0d63640 1 -- 172.21.15.16:0/3894122336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57dc14acd0 con 0x7f57dc15c0b0 2024-04-24T23:49:23.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.203+0000 7fe16ffff640 1 -- 172.21.15.16:0/88204952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe16802b740 con 0x7fe16804ffd0 2024-04-24T23:49:23.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.203+0000 7f4077695640 1 -- 172.21.15.16:0/2220237329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40680f0580 con 0x7f40680ead00 2024-04-24T23:49:23.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.203+0000 7f57caffd640 1 -- 172.21.15.16:0/3894122336 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57cc030a10 con 0x7f57dc15c0b0 2024-04-24T23:49:23.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.203+0000 7fb8fc8b9640 1 -- start start 2024-04-24T23:49:23.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.204+0000 7fb8fc8b9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f00069e0 0x7fb8f00e2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.204+0000 7fb8fc8b9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f00485d0 0x7fb8f0048a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.204+0000 7fb8fc8b9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f002b520 0x7fb8f002b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.204+0000 7fb8fc8b9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f00547c0 con 0x7fb8f00069e0 2024-04-24T23:49:23.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.204+0000 7fb8fc8b9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb8f00501a0 con 0x7fb8f00485d0 2024-04-24T23:49:23.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.204+0000 7fb8fc8b9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb8f0054940 con 0x7fb8f002b520 2024-04-24T23:49:23.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.204+0000 7f57caffd640 1 -- 172.21.15.16:0/3894122336 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57cc037040 con 0x7f57dc15c0b0 2024-04-24T23:49:23.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.204+0000 7f7807fff640 1 -- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78080ce080 msgr2=0x7f7808200ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.204+0000 7f7807fff640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78080ce080 0x7f7808200ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.204+0000 7f7807fff640 1 -- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78080ce460 msgr2=0x7f780806da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.204+0000 7f7807fff640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78080ce460 0x7f780806da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.204+0000 7f7807fff640 1 -- 172.21.15.16:0/2567191678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7808201620 con 0x7f78080b3cc0 2024-04-24T23:49:23.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7fe1577fe640 1 -- 172.21.15.16:0/88204952 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe16001a070 con 0x7fe16804ffd0 2024-04-24T23:49:23.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7f28e6ffd640 1 -- 172.21.15.16:0/436300424 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8041020 con 0x7f28f004fdb0 2024-04-24T23:49:23.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7f28e6ffd640 1 -- 172.21.15.16:0/436300424 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28e8033e00 con 0x7f28f004fdb0 2024-04-24T23:49:23.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7f7807fff640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080b3cc0 0x7f780812ac70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7800034040 tx=0x7f7800004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7f28f759e640 1 -- 172.21.15.16:0/436300424 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28f004b2d0 con 0x7f28f004fdb0 2024-04-24T23:49:23.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7fb8f759e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f002b520 0x7fb8f002b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7f28f759e640 1 -- 172.21.15.16:0/436300424 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28f00cbed0 con 0x7f28f004fdb0 2024-04-24T23:49:23.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7fe1577fe640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe15c06bf30 0x7fe15c06e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7f28e6ffd640 1 -- 172.21.15.16:0/436300424 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28e8037040 con 0x7f28f004fdb0 2024-04-24T23:49:23.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7fb8f6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f00069e0 0x7fb8f00e2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7fb8f6d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f00069e0 0x7fb8f00e2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48684/0 (socket says 172.21.15.16:48684) 2024-04-24T23:49:23.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7fb8f6d9d640 1 -- 172.21.15.16:0/620250877 learned_addr learned my addr 172.21.15.16:0/620250877 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:23.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7fe1577fe640 1 -- 172.21.15.16:0/88204952 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fe16008b490 con 0x7fe16804ffd0 2024-04-24T23:49:23.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.206+0000 7fe16effd640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe15c06bf30 0x7fe15c06e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.204+0000 7f57e0d63640 1 -- 172.21.15.16:0/3894122336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57c0005290 con 0x7f57dc15c0b0 2024-04-24T23:49:23.253 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.206+0000 7f57caffd640 1 -- 172.21.15.16:0/3894122336 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f57cc045020 con 0x7f57dc15c0b0 2024-04-24T23:49:23.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7fb37affd640 1 -- 172.21.15.16:0/1455527805 <== mon.2 v2:172.21.15.156:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb36c01a070 con 0x7fb37c04ffd0 2024-04-24T23:49:23.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.205+0000 7fb38359e640 1 -- 172.21.15.16:0/1455527805 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb37c005390 con 0x7fb37c04ffd0 2024-04-24T23:49:23.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.206+0000 7fb37affd640 1 --2- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb36406be80 0x7fb36406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.206+0000 7fb8f6d9d640 1 -- 172.21.15.16:0/620250877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f002b520 msgr2=0x7fb8f002b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.206+0000 7fb8f6d9d640 1 --2- 172.21.15.16:0/620250877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f002b520 0x7fb8f002b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.206+0000 7fb8f6d9d640 1 -- 172.21.15.16:0/620250877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f00485d0 msgr2=0x7fb8f0048a30 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T23:49:23.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.206+0000 7fb8f6d9d640 1 --2- 172.21.15.16:0/620250877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f00485d0 0x7fb8f0048a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.206+0000 7fb8f6d9d640 1 -- 172.21.15.16:0/620250877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8f002c2c0 con 0x7fb8f00069e0 2024-04-24T23:49:23.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.206+0000 7fb8f6d9d640 1 --2- 172.21.15.16:0/620250877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f00069e0 0x7fb8f00e2c80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00f8e0 tx=0x7fb8ec00fda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.206+0000 7fb38259c640 1 --2- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb36406be80 0x7fb36406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.206+0000 7fb37affd640 1 -- 172.21.15.16:0/1455527805 <== mon.2 v2:172.21.15.156:3301/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb36c056c40 con 0x7fb37c04ffd0 2024-04-24T23:49:23.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.207+0000 7f57caffd640 1 --2- 172.21.15.16:0/3894122336 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f57bc06bf00 0x7f57bc06e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.207+0000 7f57caffd640 1 -- 172.21.15.16:0/3894122336 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f57cc0abe40 con 0x7f57dc15c0b0 2024-04-24T23:49:23.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.207+0000 7f4075431640 1 --2- 172.21.15.16:0/2220237329 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f405406be80 0x7f405406e340 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f406c009c50 tx=0x7f406c009240 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.207+0000 7f57da59c640 1 --2- 172.21.15.16:0/3894122336 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f57bc06bf00 0x7f57bc06e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.207+0000 7f28e6ffd640 1 -- 172.21.15.16:0/436300424 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f28e8044020 con 0x7f28f004fdb0 2024-04-24T23:49:23.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.207+0000 7f28f759e640 1 -- 172.21.15.16:0/436300424 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28dc005290 con 0x7f28f004fdb0 2024-04-24T23:49:23.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.208+0000 7fb38259c640 1 --2- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb36406be80 0x7fb36406e340 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0544b0 tx=0x7fb3840ab000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.208+0000 7fe16effd640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe15c06bf30 0x7fe15c06e3f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe164034000 tx=0x7fe16402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.208+0000 7f57da59c640 1 --2- 172.21.15.16:0/3894122336 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f57bc06bf00 0x7f57bc06e3c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f57d0001700 tx=0x7f57d0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.208+0000 7f28e6ffd640 1 --2- 172.21.15.16:0/436300424 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f28d806bf00 0x7f28d806e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.208+0000 7f28e6ffd640 1 -- 172.21.15.16:0/436300424 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f28e80abda0 con 0x7f28f004fdb0 2024-04-24T23:49:23.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.208+0000 7f28f5d9b640 1 --2- 172.21.15.16:0/436300424 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f28d806bf00 0x7f28d806e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.209+0000 7f28f5d9b640 1 --2- 172.21.15.16:0/436300424 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f28d806bf00 0x7f28d806e3c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f28f00e53e0 tx=0x7f28ec009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.211+0000 7f4065ffb640 1 -- 172.21.15.16:0/2220237329 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f40700b7040 con 0x7f40680ead00 2024-04-24T23:49:23.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.216+0000 7fb37affd640 1 -- 172.21.15.16:0/1455527805 <== mon.2 v2:172.21.15.156:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb36c058ba0 con 0x7fb37c04ffd0 2024-04-24T23:49:23.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.217+0000 7f28e6ffd640 1 -- 172.21.15.16:0/436300424 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f28e804b030 con 0x7f28f004fdb0 2024-04-24T23:49:23.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.219+0000 7fe1577fe640 1 -- 172.21.15.16:0/88204952 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe160059490 con 0x7fe16804ffd0 2024-04-24T23:49:23.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.228+0000 7f57caffd640 1 -- 172.21.15.16:0/3894122336 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f57cc04c030 con 0x7f57dc15c0b0 2024-04-24T23:49:23.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.228+0000 7fb8df7fe640 1 -- 172.21.15.16:0/620250877 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec0053a0 con 0x7fb8f00069e0 2024-04-24T23:49:23.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.228+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/620250877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8f00e5590 con 0x7fb8f00069e0 2024-04-24T23:49:23.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.228+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/620250877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8f00e5b50 con 0x7fb8f00069e0 2024-04-24T23:49:23.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.229+0000 7fb8df7fe640 1 -- 172.21.15.16:0/620250877 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8ec018070 con 0x7fb8f00069e0 2024-04-24T23:49:23.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.229+0000 7fb8df7fe640 1 -- 172.21.15.16:0/620250877 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec00d930 con 0x7fb8f00069e0 2024-04-24T23:49:23.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.229+0000 7f7804ff9640 1 -- 172.21.15.16:0/2567191678 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7800036050 con 0x7f78080b3cc0 2024-04-24T23:49:23.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.229+0000 7f7804ff9640 1 -- 172.21.15.16:0/2567191678 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7800042070 con 0x7f78080b3cc0 2024-04-24T23:49:23.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.229+0000 7f7804ff9640 1 -- 172.21.15.16:0/2567191678 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780003d450 con 0x7f78080b3cc0 2024-04-24T23:49:23.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.230+0000 7f780e407640 1 -- 172.21.15.16:0/2567191678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7808201900 con 0x7f78080b3cc0 2024-04-24T23:49:23.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.230+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/620250877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8d8005290 con 0x7fb8f00069e0 2024-04-24T23:49:23.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.230+0000 7f780e407640 1 -- 172.21.15.16:0/2567191678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7808128ad0 con 0x7f78080b3cc0 2024-04-24T23:49:23.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.231+0000 7f780e407640 1 -- 172.21.15.16:0/2567191678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78080731f0 con 0x7f78080b3cc0 2024-04-24T23:49:23.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.233+0000 7fb8df7fe640 1 -- 172.21.15.16:0/620250877 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb8ec00dad0 con 0x7fb8f00069e0 2024-04-24T23:49:23.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.233+0000 7fb8df7fe640 1 --2- 172.21.15.16:0/620250877 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb8e406be30 0x7fb8e406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.234+0000 7fb8df7fe640 1 -- 172.21.15.16:0/620250877 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fb8ec08bdf0 con 0x7fb8f00069e0 2024-04-24T23:49:23.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.233+0000 7f7804ff9640 1 -- 172.21.15.16:0/2567191678 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f780003d5f0 con 0x7f78080b3cc0 2024-04-24T23:49:23.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.233+0000 7f7804ff9640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f77f406bf00 0x7f77f406e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:23.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.234+0000 7f7804ff9640 1 -- 172.21.15.16:0/2567191678 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f78000acda0 con 0x7f78080b3cc0 2024-04-24T23:49:23.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.234+0000 7f78077fe640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f77f406bf00 0x7f77f406e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.234+0000 7fb8f659c640 1 --2- 172.21.15.16:0/620250877 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb8e406be30 0x7fb8e406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:23.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.237+0000 7fb8df7fe640 1 -- 172.21.15.16:0/620250877 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb8ec05a440 con 0x7fb8f00069e0 2024-04-24T23:49:23.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.238+0000 7fb8f659c640 1 --2- 172.21.15.16:0/620250877 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb8e406be30 0x7fb8e406e2f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb8e80088a0 tx=0x7fb8e80087f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.238+0000 7f78077fe640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f77f406bf00 0x7f77f406e3c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7808130b80 tx=0x7f77fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:23.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.246+0000 7f7804ff9640 1 -- 172.21.15.16:0/2567191678 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f780007aef0 con 0x7f78080b3cc0 2024-04-24T23:49:23.560 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.561+0000 7fbcea4da640 1 -- 172.21.15.16:0/4014661529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fbcc400dd90 con 0x7fbce4144120 2024-04-24T23:49:23.561 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.562+0000 7fbce0ff9640 1 -- 172.21.15.16:0/4014661529 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbcdc07eac0 con 0x7fbce4144120 2024-04-24T23:49:23.562 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2024-04-24T23:49:23.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.565+0000 7fbcca7fc640 1 -- 172.21.15.16:0/4014661529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbccc0049b0 msgr2=0x7fbccc004e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.566+0000 7fbcca7fc640 1 --2- 172.21.15.16:0/4014661529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbccc0049b0 0x7fbccc004e60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbce413a880 tx=0x7fbcd8009060 comp rx=0 tx=0).stop 2024-04-24T23:49:23.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.566+0000 7fbcca7fc640 1 -- 172.21.15.16:0/4014661529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce4144120 msgr2=0x7fbce413d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.566+0000 7fbcca7fc640 1 --2- 172.21.15.16:0/4014661529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce4144120 0x7fbce413d010 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbcdc031000 tx=0x7fbcdc004a40 comp rx=0 tx=0).stop 2024-04-24T23:49:23.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.566+0000 7fbcca7fc640 1 -- 172.21.15.16:0/4014661529 shutdown_connections 2024-04-24T23:49:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.566+0000 7fbcca7fc640 1 --2- 172.21.15.16:0/4014661529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbccc0049b0 0x7fbccc004e60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.566+0000 7fbcca7fc640 1 --2- 172.21.15.16:0/4014661529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbce4144120 0x7fbce413d010 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.566+0000 7fbcca7fc640 1 --2- 172.21.15.16:0/4014661529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbce40cf530 0x7fbce413cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.566+0000 7fbcca7fc640 1 --2- 172.21.15.16:0/4014661529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbce4072b50 0x7fbce40aa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.567+0000 7fbcca7fc640 1 -- 172.21.15.16:0/4014661529 >> 172.21.15.16:0/4014661529 conn(0x7fbce40cd770 msgr2=0x7fbce406cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.567+0000 7fbcca7fc640 1 -- 172.21.15.16:0/4014661529 shutdown_connections 2024-04-24T23:49:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.567+0000 7fbcca7fc640 1 -- 172.21.15.16:0/4014661529 wait complete. 2024-04-24T23:49:23.590 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574857 for osd.6 2024-04-24T23:49:23.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.636+0000 7fe16ffff640 1 -- 172.21.15.16:0/88204952 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fe168005370 con 0x7fe16804ffd0 2024-04-24T23:49:23.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.637+0000 7fe1577fe640 1 -- 172.21.15.16:0/88204952 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe16005d790 con 0x7fe16804ffd0 2024-04-24T23:49:23.640 INFO:teuthology.orchestra.run.smithi016.stdout:55834574860 2024-04-24T23:49:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.641+0000 7fe16ffff640 1 -- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe15c06bf30 msgr2=0x7fe15c06e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.641+0000 7fe16ffff640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe15c06bf30 0x7fe15c06e3f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe164034000 tx=0x7fe16402e000 comp rx=0 tx=0).stop 2024-04-24T23:49:23.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.641+0000 7fe16ffff640 1 -- 172.21.15.16:0/88204952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16804ffd0 msgr2=0x7fe16802b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.641+0000 7fe16ffff640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16804ffd0 0x7fe16802b960 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fe16000e9e0 tx=0x7fe16000eea0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.642+0000 7fe16ffff640 1 -- 172.21.15.16:0/88204952 shutdown_connections 2024-04-24T23:49:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.642+0000 7fe16ffff640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe15c06bf30 0x7fe15c06e3f0 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe164034000 tx=0x7fe16402e000 comp rx=0 tx=0).stop 2024-04-24T23:49:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.642+0000 7fe16ffff640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe1680ead00 0x7fe1680e9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.642+0000 7fe16ffff640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe16804ffd0 0x7fe16802b960 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.642+0000 7fe16ffff640 1 --2- 172.21.15.16:0/88204952 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe168038110 0x7fe16802e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.642+0000 7fe16ffff640 1 -- 172.21.15.16:0/88204952 >> 172.21.15.16:0/88204952 conn(0x7fe168009fb0 msgr2=0x7fe16804af90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.642+0000 7fe16ffff640 1 -- 172.21.15.16:0/88204952 shutdown_connections 2024-04-24T23:49:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.642+0000 7fe16ffff640 1 -- 172.21.15.16:0/88204952 wait complete. 2024-04-24T23:49:23.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.642+0000 7f4077695640 1 -- 172.21.15.16:0/2220237329 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f4068005390 con 0x7f40680ead00 2024-04-24T23:49:23.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.643+0000 7f4065ffb640 1 -- 172.21.15.16:0/2220237329 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f40700ad030 con 0x7f40680ead00 2024-04-24T23:49:23.643 INFO:teuthology.orchestra.run.smithi016.stdout:55834574859 2024-04-24T23:49:23.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.646+0000 7f4077695640 1 -- 172.21.15.16:0/2220237329 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f405406be80 msgr2=0x7f405406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.647+0000 7f4077695640 1 --2- 172.21.15.16:0/2220237329 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f405406be80 0x7f405406e340 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f406c009c50 tx=0x7f406c009240 comp rx=0 tx=0).stop 2024-04-24T23:49:23.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.647+0000 7f4077695640 1 -- 172.21.15.16:0/2220237329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40680ead00 msgr2=0x7f4068188f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.647+0000 7f4077695640 1 --2- 172.21.15.16:0/2220237329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40680ead00 0x7f4068188f50 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f407007a980 tx=0x7f407004fc00 comp rx=0 tx=0).stop 2024-04-24T23:49:23.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.647+0000 7f4077695640 1 -- 172.21.15.16:0/2220237329 shutdown_connections 2024-04-24T23:49:23.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.647+0000 7f4077695640 1 --2- 172.21.15.16:0/2220237329 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f405406be80 0x7f405406e340 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f406c009c50 tx=0x7f406c009240 comp rx=0 tx=0).stop 2024-04-24T23:49:23.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.647+0000 7f4077695640 1 --2- 172.21.15.16:0/2220237329 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40680ead00 0x7f4068188f50 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.647+0000 7f4077695640 1 --2- 172.21.15.16:0/2220237329 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f406804ffd0 0x7f406804ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.648+0000 7f4077695640 1 --2- 172.21.15.16:0/2220237329 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4068038110 0x7f40680365b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.648+0000 7f4077695640 1 -- 172.21.15.16:0/2220237329 >> 172.21.15.16:0/2220237329 conn(0x7f4068009fb0 msgr2=0x7f406804ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.648+0000 7f4077695640 1 -- 172.21.15.16:0/2220237329 shutdown_connections 2024-04-24T23:49:23.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.648+0000 7f4077695640 1 -- 172.21.15.16:0/2220237329 wait complete. 2024-04-24T23:49:23.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.653+0000 7fb38359e640 1 -- 172.21.15.16:0/1455527805 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fb37c048cf0 con 0x7fb37c04ffd0 2024-04-24T23:49:23.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.654+0000 7fb37affd640 1 -- 172.21.15.16:0/1455527805 <== mon.2 v2:172.21.15.156:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb36c0616d0 con 0x7fb37c04ffd0 2024-04-24T23:49:23.653 INFO:teuthology.orchestra.run.smithi016.stdout:55834574860 2024-04-24T23:49:23.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.657+0000 7fb378ff9640 1 -- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb36406be80 msgr2=0x7fb36406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.657+0000 7fb378ff9640 1 --2- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb36406be80 0x7fb36406e340 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0544b0 tx=0x7fb3840ab000 comp rx=0 tx=0).stop 2024-04-24T23:49:23.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.657+0000 7fb378ff9640 1 -- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb37c04ffd0 msgr2=0x7fb37c04e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.657+0000 7fb378ff9640 1 --2- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb37c04ffd0 0x7fb37c04e550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb36c00e600 tx=0x7fb36c00eac0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.658+0000 7fb378ff9640 1 -- 172.21.15.16:0/1455527805 shutdown_connections 2024-04-24T23:49:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.658+0000 7fb378ff9640 1 --2- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb36406be80 0x7fb36406e340 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.658+0000 7fb378ff9640 1 --2- 172.21.15.16:0/1455527805 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb37c0ead00 0x7fb37c045a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.658+0000 7fb378ff9640 1 --2- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb37c04ffd0 0x7fb37c04e550 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.658+0000 7fb378ff9640 1 --2- 172.21.15.16:0/1455527805 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb37c038110 0x7fb37c0dcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.658+0000 7fb378ff9640 1 -- 172.21.15.16:0/1455527805 >> 172.21.15.16:0/1455527805 conn(0x7fb37c009fb0 msgr2=0x7fb37c004930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.659+0000 7fb378ff9640 1 -- 172.21.15.16:0/1455527805 shutdown_connections 2024-04-24T23:49:23.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.659+0000 7fb378ff9640 1 -- 172.21.15.16:0/1455527805 wait complete. 2024-04-24T23:49:23.660 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574860 for osd.3 2024-04-24T23:49:23.660 DEBUG:teuthology.parallel:result is None 2024-04-24T23:49:23.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.665+0000 7f28f759e640 1 -- 172.21.15.16:0/436300424 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f28dc005580 con 0x7f28f004fdb0 2024-04-24T23:49:23.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.665+0000 7f57e0d63640 1 -- 172.21.15.16:0/3894122336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f57c0005580 con 0x7f57dc15c0b0 2024-04-24T23:49:23.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.665+0000 7f28e6ffd640 1 -- 172.21.15.16:0/436300424 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f28e8030e20 con 0x7f28f004fdb0 2024-04-24T23:49:23.667 INFO:teuthology.orchestra.run.smithi016.stdout:55834574859 2024-04-24T23:49:23.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.666+0000 7f57caffd640 1 -- 172.21.15.16:0/3894122336 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f57cc07a520 con 0x7f57dc15c0b0 2024-04-24T23:49:23.668 INFO:teuthology.orchestra.run.smithi016.stdout:55834574859 2024-04-24T23:49:23.670 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.5 2024-04-24T23:49:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f28f759e640 1 -- 172.21.15.16:0/436300424 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f28d806bf00 msgr2=0x7f28d806e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f28f759e640 1 --2- 172.21.15.16:0/436300424 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f28d806bf00 0x7f28d806e3c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f28f00e53e0 tx=0x7f28ec009040 comp rx=0 tx=0).stop 2024-04-24T23:49:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f28f759e640 1 -- 172.21.15.16:0/436300424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28f004fdb0 msgr2=0x7f28f00cdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f28f759e640 1 --2- 172.21.15.16:0/436300424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28f004fdb0 0x7f28f00cdef0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f28e8000c00 tx=0x7f28e80046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.672 DEBUG:teuthology.parallel:result is None 2024-04-24T23:49:23.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f57e0d63640 1 -- 172.21.15.16:0/3894122336 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f57bc06bf00 msgr2=0x7f57bc06e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f57e0d63640 1 --2- 172.21.15.16:0/3894122336 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f57bc06bf00 0x7f57bc06e3c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f57d0001700 tx=0x7f57d0009040 comp rx=0 tx=0).stop 2024-04-24T23:49:23.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f57e0d63640 1 -- 172.21.15.16:0/3894122336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57dc15c0b0 msgr2=0x7f57dc150360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f28f759e640 1 -- 172.21.15.16:0/436300424 shutdown_connections 2024-04-24T23:49:23.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f57e0d63640 1 --2- 172.21.15.16:0/3894122336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57dc15c0b0 0x7f57dc150360 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f57cc031000 tx=0x7f57cc0046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f28f759e640 1 --2- 172.21.15.16:0/436300424 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f28d806bf00 0x7f28d806e3c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f28f759e640 1 --2- 172.21.15.16:0/436300424 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f28f00e3260 0x7f28f00e3d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f28f759e640 1 --2- 172.21.15.16:0/436300424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28f0053380 0x7f28f00e3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f28f759e640 1 --2- 172.21.15.16:0/436300424 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f28f004fdb0 0x7f28f00cdef0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f28f759e640 1 -- 172.21.15.16:0/436300424 >> 172.21.15.16:0/436300424 conn(0x7f28f0009fb0 msgr2=0x7f28f0004930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f57e0d63640 1 -- 172.21.15.16:0/3894122336 shutdown_connections 2024-04-24T23:49:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f57e0d63640 1 --2- 172.21.15.16:0/3894122336 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f57bc06bf00 0x7f57bc06e3c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f57e0d63640 1 --2- 172.21.15.16:0/3894122336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57dc15c490 0x7f57dc14d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f57e0d63640 1 --2- 172.21.15.16:0/3894122336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57dc15c0b0 0x7f57dc150360 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f57e0d63640 1 --2- 172.21.15.16:0/3894122336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57dc0cf220 0x7f57dc14bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f57e0d63640 1 -- 172.21.15.16:0/3894122336 >> 172.21.15.16:0/3894122336 conn(0x7f57dc0cdcc0 msgr2=0x7f57dc0b1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f28f759e640 1 -- 172.21.15.16:0/436300424 shutdown_connections 2024-04-24T23:49:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f57e0d63640 1 -- 172.21.15.16:0/3894122336 shutdown_connections 2024-04-24T23:49:23.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f28f759e640 1 -- 172.21.15.16:0/436300424 wait complete. 2024-04-24T23:49:23.679 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.669+0000 7f57e0d63640 1 -- 172.21.15.16:0/3894122336 wait complete. 2024-04-24T23:49:23.680 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574859 got 55834574860 for osd.2 2024-04-24T23:49:23.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.677+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/620250877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fb8d8005850 con 0x7fb8f00069e0 2024-04-24T23:49:23.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.678+0000 7fb8df7fe640 1 -- 172.21.15.16:0/620250877 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb8ec059d40 con 0x7fb8f00069e0 2024-04-24T23:49:23.682 DEBUG:teuthology.parallel:result is None 2024-04-24T23:49:23.682 INFO:teuthology.orchestra.run.smithi016.stdout:55834574859 2024-04-24T23:49:23.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.681+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/620250877 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb8e406be30 msgr2=0x7fb8e406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.681+0000 7fb8fc8b9640 1 --2- 172.21.15.16:0/620250877 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb8e406be30 0x7fb8e406e2f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb8e80088a0 tx=0x7fb8e80087f0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.681+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/620250877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f00069e0 msgr2=0x7fb8f00e2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.681+0000 7fb8fc8b9640 1 --2- 172.21.15.16:0/620250877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f00069e0 0x7fb8f00e2c80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00f8e0 tx=0x7fb8ec00fda0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.681+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/620250877 shutdown_connections 2024-04-24T23:49:23.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.681+0000 7fb8fc8b9640 1 --2- 172.21.15.16:0/620250877 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb8e406be30 0x7fb8e406e2f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.681+0000 7fb8fc8b9640 1 --2- 172.21.15.16:0/620250877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb8f002b520 0x7fb8f002b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.681+0000 7fb8fc8b9640 1 --2- 172.21.15.16:0/620250877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb8f00485d0 0x7fb8f0048a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.681+0000 7fb8fc8b9640 1 --2- 172.21.15.16:0/620250877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb8f00069e0 0x7fb8f00e2c80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.681+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/620250877 >> 172.21.15.16:0/620250877 conn(0x7fb8f0009fb0 msgr2=0x7fb8f003a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.681+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/620250877 shutdown_connections 2024-04-24T23:49:23.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.681+0000 7fb8fc8b9640 1 -- 172.21.15.16:0/620250877 wait complete. 2024-04-24T23:49:23.687 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.7 2024-04-24T23:49:23.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.682+0000 7f780e407640 1 -- 172.21.15.16:0/2567191678 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f780804f5e0 con 0x7f78080b3cc0 2024-04-24T23:49:23.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.683+0000 7f7804ff9640 1 -- 172.21.15.16:0/2567191678 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f780007f1f0 con 0x7f78080b3cc0 2024-04-24T23:49:23.688 DEBUG:teuthology.parallel:result is None 2024-04-24T23:49:23.688 INFO:teuthology.orchestra.run.smithi016.stdout:55834574859 2024-04-24T23:49:23.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.686+0000 7f780e407640 1 -- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f77f406bf00 msgr2=0x7f77f406e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.686+0000 7f780e407640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f77f406bf00 0x7f77f406e3c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7808130b80 tx=0x7f77fc009040 comp rx=0 tx=0).stop 2024-04-24T23:49:23.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.686+0000 7f780e407640 1 -- 172.21.15.16:0/2567191678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080b3cc0 msgr2=0x7f780812ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:23.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.686+0000 7f780e407640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080b3cc0 0x7f780812ac70 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7800034040 tx=0x7f7800004a40 comp rx=0 tx=0).stop 2024-04-24T23:49:23.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.686+0000 7f780e407640 1 -- 172.21.15.16:0/2567191678 shutdown_connections 2024-04-24T23:49:23.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.686+0000 7f780e407640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f77f406bf00 0x7f77f406e3c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.686+0000 7f780e407640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78080ce460 0x7f780806da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.686+0000 7f780e407640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f78080ce080 0x7f7808200ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.686+0000 7f780e407640 1 --2- 172.21.15.16:0/2567191678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78080b3cc0 0x7f780812ac70 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:23.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.686+0000 7f780e407640 1 -- 172.21.15.16:0/2567191678 >> 172.21.15.16:0/2567191678 conn(0x7f78080b2000 msgr2=0x7f780815bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:23.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.686+0000 7f780e407640 1 -- 172.21.15.16:0/2567191678 shutdown_connections 2024-04-24T23:49:23.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:23.686+0000 7f780e407640 1 -- 172.21.15.16:0/2567191678 wait complete. 2024-04-24T23:49:23.692 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.1 2024-04-24T23:49:23.693 DEBUG:teuthology.parallel:result is None 2024-04-24T23:49:23.695 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.4 2024-04-24T23:49:23.695 DEBUG:teuthology.parallel:result is None 2024-04-24T23:49:23.698 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.0 2024-04-24T23:49:23.698 DEBUG:teuthology.parallel:result is None 2024-04-24T23:49:23.917 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:49:23.920 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:49:24.590 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-04-24T23:49:24.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.693+0000 7f0593fe7640 1 Processor -- start 2024-04-24T23:49:24.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.693+0000 7f0593fe7640 1 -- start start 2024-04-24T23:49:24.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.694+0000 7f0593fe7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f058c071a00 0x7f058c071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.694+0000 7f0593fe7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f058c144740 0x7f058c15c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.694+0000 7f0593fe7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f058c072320 con 0x7f058c15c8e0 2024-04-24T23:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.694+0000 7f0593fe7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f058c15c1f0 con 0x7f058c071a00 2024-04-24T23:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.694+0000 7f0593fe7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f058c0cf3a0 con 0x7f058c144740 2024-04-24T23:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.694+0000 7f0592584640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f058c15c8e0 0x7f058c0ce7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50736/0 (socket says 172.21.15.16:50736) 2024-04-24T23:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.694+0000 7f0592584640 1 -- 172.21.15.16:0/346360059 learned_addr learned my addr 172.21.15.16:0/346360059 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0591582640 1 --2- 172.21.15.16:0/346360059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f058c144740 0x7f058c15c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0591d83640 1 --2- 172.21.15.16:0/346360059 >> v2:172.21.15.156:3300/0 conn(0x7f058c071a00 0x7f058c071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 -- 172.21.15.16:0/346360059 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3798739739 0 0) 0x7f058c072320 con 0x7f058c15c8e0 2024-04-24T23:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 -- 172.21.15.16:0/346360059 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0578003520 con 0x7f058c15c8e0 2024-04-24T23:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 -- 172.21.15.16:0/346360059 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1536360119 0 0) 0x7f0578003520 con 0x7f058c15c8e0 2024-04-24T23:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 -- 172.21.15.16:0/346360059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f058c144740 msgr2=0x7f058c15c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:24.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 --2- 172.21.15.16:0/346360059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f058c144740 0x7f058c15c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 -- 172.21.15.16:0/346360059 >> v2:172.21.15.156:3300/0 conn(0x7f058c071a00 msgr2=0x7f058c071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 --2- 172.21.15.16:0/346360059 >> v2:172.21.15.156:3300/0 conn(0x7f058c071a00 0x7f058c071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 -- 172.21.15.16:0/346360059 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058c0cc8d0 con 0x7f058c15c8e0 2024-04-24T23:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 -- 172.21.15.16:0/346360059 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f05840019f0 con 0x7f058c15c8e0 2024-04-24T23:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 -- 172.21.15.16:0/346360059 >> v1:172.21.15.16:6789/0 conn(0x7f058c15c8e0 legacy=0x7f058c0ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 --2- 172.21.15.16:0/346360059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05780039b0 0x7f0578003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 --2- 172.21.15.16:0/346360059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0578004800 0x7f0578004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 --2- 172.21.15.16:0/346360059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05780051c0 0x7f057800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 -- 172.21.15.16:0/346360059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f058c0cf3a0 con 0x7f0578004800 2024-04-24T23:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 -- 172.21.15.16:0/346360059 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0578004580 con 0x7f05780039b0 2024-04-24T23:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0590d81640 1 -- 172.21.15.16:0/346360059 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f057800bf00 con 0x7f05780051c0 2024-04-24T23:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0591582640 1 --2- 172.21.15.16:0/346360059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0578004800 0x7f0578004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0591d83640 1 --2- 172.21.15.16:0/346360059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05780039b0 0x7f0578003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0592584640 1 --2- 172.21.15.16:0/346360059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05780051c0 0x7f057800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0591582640 1 -- 172.21.15.16:0/346360059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05780051c0 msgr2=0x7f057800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0591582640 1 --2- 172.21.15.16:0/346360059 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f05780051c0 0x7f057800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0591582640 1 -- 172.21.15.16:0/346360059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05780039b0 msgr2=0x7f0578003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0591582640 1 --2- 172.21.15.16:0/346360059 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f05780039b0 0x7f0578003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.695+0000 7f0591582640 1 -- 172.21.15.16:0/346360059 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0578003520 con 0x7f0578004800 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.696+0000 7f0591582640 1 --2- 172.21.15.16:0/346360059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0578004800 0x7f0578004c80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0580009ac0 tx=0x7f0578005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b9425fca86b3d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.696+0000 7f0590d81640 1 -- 172.21.15.16:0/346360059 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580036070 con 0x7f0578004800 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.696+0000 7f0590d81640 1 -- 172.21.15.16:0/346360059 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0580031400 con 0x7f0578004800 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.696+0000 7f0590d81640 1 -- 172.21.15.16:0/346360059 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0580009690 con 0x7f0578004800 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.697+0000 7f0593fe7640 1 -- 172.21.15.16:0/346360059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0578004800 msgr2=0x7f0578004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.697+0000 7f0593fe7640 1 --2- 172.21.15.16:0/346360059 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0578004800 0x7f0578004c80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f0580009ac0 tx=0x7f0578005830 comp rx=0 tx=0).stop 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.697+0000 7f0592584640 1 -- 172.21.15.16:0/346360059 reap_dead start 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.697+0000 7f0593fe7640 1 -- 172.21.15.16:0/346360059 shutdown_connections 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.697+0000 7f0593fe7640 1 -- 172.21.15.16:0/346360059 >> 172.21.15.16:0/346360059 conn(0x7f058c0b2000 msgr2=0x7f058c0cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.697+0000 7f0593fe7640 1 -- 172.21.15.16:0/346360059 shutdown_connections 2024-04-24T23:49:24.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.697+0000 7f0593fe7640 1 -- 172.21.15.16:0/346360059 wait complete. 2024-04-24T23:49:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.698+0000 7f0593fe7640 1 Processor -- start 2024-04-24T23:49:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.698+0000 7f0593fe7640 1 -- start start 2024-04-24T23:49:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.699+0000 7f0593fe7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0578004800 0x7f058c159ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.699+0000 7f0593fe7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f058c143d20 0x7f058c141b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.699+0000 7f0593fe7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f058c0709f0 0x7f058c070ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.699+0000 7f0593fe7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f058c0cd170 con 0x7f058c0709f0 2024-04-24T23:49:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.699+0000 7f0593fe7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f058c0cce90 con 0x7f0578004800 2024-04-24T23:49:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.699+0000 7f0593fe7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f058c0ceed0 con 0x7f058c143d20 2024-04-24T23:49:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.699+0000 7f0592584640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f058c0709f0 0x7f058c070ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.699+0000 7f0592584640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f058c0709f0 0x7f058c070ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48704/0 (socket says 172.21.15.16:48704) 2024-04-24T23:49:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.699+0000 7f0592584640 1 -- 172.21.15.16:0/3146010025 learned_addr learned my addr 172.21.15.16:0/3146010025 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.699+0000 7f0591d83640 1 --2- 172.21.15.16:0/3146010025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0578004800 0x7f058c159ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.699+0000 7f0591582640 1 --2- 172.21.15.16:0/3146010025 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f058c143d20 0x7f058c141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.700+0000 7f0592584640 1 -- 172.21.15.16:0/3146010025 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f058c143d20 msgr2=0x7f058c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.700+0000 7f0592584640 1 --2- 172.21.15.16:0/3146010025 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f058c143d20 0x7f058c141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.700+0000 7f0592584640 1 -- 172.21.15.16:0/3146010025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0578004800 msgr2=0x7f058c159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.700+0000 7f0592584640 1 --2- 172.21.15.16:0/3146010025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0578004800 0x7f058c159ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.700+0000 7f0592584640 1 -- 172.21.15.16:0/3146010025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f058c13ef10 con 0x7f058c0709f0 2024-04-24T23:49:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.700+0000 7f0592584640 1 --2- 172.21.15.16:0/3146010025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f058c0709f0 0x7f058c070ea0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f05840013d0 tx=0x7f058400a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.700+0000 7f057e7fc640 1 -- 172.21.15.16:0/3146010025 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f058400ad50 con 0x7f058c0709f0 2024-04-24T23:49:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.700+0000 7f057e7fc640 1 -- 172.21.15.16:0/3146010025 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0584019070 con 0x7f058c0709f0 2024-04-24T23:49:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.700+0000 7f057e7fc640 1 -- 172.21.15.16:0/3146010025 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0584013630 con 0x7f058c0709f0 2024-04-24T23:49:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.700+0000 7f0593fe7640 1 -- 172.21.15.16:0/3146010025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f058c13e150 con 0x7f058c0709f0 2024-04-24T23:49:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.700+0000 7f0593fe7640 1 -- 172.21.15.16:0/3146010025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f058c13e710 con 0x7f058c0709f0 2024-04-24T23:49:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.701+0000 7f0593fe7640 1 -- 172.21.15.16:0/3146010025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f058c0731f0 con 0x7f058c0709f0 2024-04-24T23:49:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.701+0000 7f057e7fc640 1 -- 172.21.15.16:0/3146010025 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f05840137d0 con 0x7f058c0709f0 2024-04-24T23:49:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.702+0000 7f057e7fc640 1 --2- 172.21.15.16:0/3146010025 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f057006bd70 0x7f057006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.702+0000 7f057e7fc640 1 -- 172.21.15.16:0/3146010025 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f058408b910 con 0x7f058c0709f0 2024-04-24T23:49:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.702+0000 7f0591d83640 1 --2- 172.21.15.16:0/3146010025 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f057006bd70 0x7f057006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.703+0000 7f0591d83640 1 --2- 172.21.15.16:0/3146010025 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f057006bd70 0x7f057006e230 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f058c06d110 tx=0x7f0588008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:24.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.706+0000 7f057e7fc640 1 -- 172.21.15.16:0/3146010025 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0584059f80 con 0x7f058c0709f0 2024-04-24T23:49:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.892+0000 7f0593fe7640 1 -- 172.21.15.16:0/3146010025 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f058c13da60 con 0x7f058c0709f0 2024-04-24T23:49:24.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.892+0000 7f057e7fc640 1 -- 172.21.15.16:0/3146010025 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0584059880 con 0x7f058c0709f0 2024-04-24T23:49:24.892 INFO:teuthology.orchestra.run.smithi016.stdout:55834574859 2024-04-24T23:49:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.894+0000 7f0593fe7640 1 -- 172.21.15.16:0/3146010025 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f057006bd70 msgr2=0x7f057006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.894+0000 7f0593fe7640 1 --2- 172.21.15.16:0/3146010025 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f057006bd70 0x7f057006e230 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f058c06d110 tx=0x7f0588008040 comp rx=0 tx=0).stop 2024-04-24T23:49:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.894+0000 7f0593fe7640 1 -- 172.21.15.16:0/3146010025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f058c0709f0 msgr2=0x7f058c070ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.894+0000 7f0593fe7640 1 --2- 172.21.15.16:0/3146010025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f058c0709f0 0x7f058c070ea0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f05840013d0 tx=0x7f058400a750 comp rx=0 tx=0).stop 2024-04-24T23:49:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.894+0000 7f0593fe7640 1 -- 172.21.15.16:0/3146010025 shutdown_connections 2024-04-24T23:49:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.894+0000 7f0593fe7640 1 --2- 172.21.15.16:0/3146010025 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f057006bd70 0x7f057006e230 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.894+0000 7f0593fe7640 1 --2- 172.21.15.16:0/3146010025 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f058c0709f0 0x7f058c070ea0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.894+0000 7f0593fe7640 1 --2- 172.21.15.16:0/3146010025 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f058c143d20 0x7f058c141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.894+0000 7f0593fe7640 1 --2- 172.21.15.16:0/3146010025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0578004800 0x7f058c159ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.894+0000 7f0593fe7640 1 -- 172.21.15.16:0/3146010025 >> 172.21.15.16:0/3146010025 conn(0x7f058c0b2000 msgr2=0x7f058c06dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.894+0000 7f0593fe7640 1 -- 172.21.15.16:0/3146010025 shutdown_connections 2024-04-24T23:49:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:24.894+0000 7f0593fe7640 1 -- 172.21.15.16:0/3146010025 wait complete. 2024-04-24T23:49:24.904 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574858 got 55834574859 for osd.6 2024-04-24T23:49:24.904 DEBUG:teuthology.parallel:result is None 2024-04-24T23:49:24.904 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T23:49:24.904 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:49:25.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.006+0000 7f3ab2c8c640 1 Processor -- start 2024-04-24T23:49:25.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.006+0000 7f3ab2c8c640 1 -- start start 2024-04-24T23:49:25.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.007+0000 7f3ab2c8c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3aac144b20 0x7f3aac144f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.007+0000 7f3ab2c8c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3aac0ced90 0x7f3aac06da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.007+0000 7f3ab2c8c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aac145440 con 0x7f3aac1457b0 2024-04-24T23:49:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.007+0000 7f3ab2c8c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f3aac15fe90 con 0x7f3aac0ced90 2024-04-24T23:49:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.007+0000 7f3ab2c8c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f3aac160170 con 0x7f3aac144b20 2024-04-24T23:49:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.007+0000 7f3ab1229640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3aac0ced90 0x7f3aac06da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.007+0000 7f3ab1229640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3aac0ced90 0x7f3aac06da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48714/0 (socket says 172.21.15.16:48714) 2024-04-24T23:49:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.007+0000 7f3ab1229640 1 -- 172.21.15.16:0/793375010 learned_addr learned my addr 172.21.15.16:0/793375010 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.007+0000 7f3ab0a28640 1 --2- 172.21.15.16:0/793375010 >> v2:172.21.15.156:3300/0 conn(0x7f3aac144b20 0x7f3aac144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.007+0000 7f3ab1229640 1 -- 172.21.15.16:0/793375010 >> v2:172.21.15.156:3300/0 conn(0x7f3aac144b20 msgr2=0x7f3aac144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.007+0000 7f3ab1229640 1 --2- 172.21.15.16:0/793375010 >> v2:172.21.15.156:3300/0 conn(0x7f3aac144b20 0x7f3aac144f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.007+0000 7f3ab1229640 1 -- 172.21.15.16:0/793375010 >> v1:172.21.15.156:6789/0 conn(0x7f3aac1457b0 legacy=0x7f3aac0ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.007+0000 7f3ab1229640 1 -- 172.21.15.16:0/793375010 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aac160450 con 0x7f3aac0ced90 2024-04-24T23:49:25.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.008+0000 7f3ab1229640 1 --2- 172.21.15.16:0/793375010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac0ced90 0x7f3aac06da40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3aac0cc410 tx=0x7f3aa402a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520468127688e793 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.008+0000 7f3aab7fe640 1 -- 172.21.15.16:0/793375010 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4036070 con 0x7f3aac0ced90 2024-04-24T23:49:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.008+0000 7f3aab7fe640 1 -- 172.21.15.16:0/793375010 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aa4031600 con 0x7f3aac0ced90 2024-04-24T23:49:25.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.008+0000 7f3aab7fe640 1 -- 172.21.15.16:0/793375010 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa40319d0 con 0x7f3aac0ced90 2024-04-24T23:49:25.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.008+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/793375010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac0ced90 msgr2=0x7f3aac06da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.008+0000 7f3ab2c8c640 1 --2- 172.21.15.16:0/793375010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac0ced90 0x7f3aac06da40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3aac0cc410 tx=0x7f3aa402a5b0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.009+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/793375010 shutdown_connections 2024-04-24T23:49:25.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.009+0000 7f3ab2c8c640 1 --2- 172.21.15.16:0/793375010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac0ced90 0x7f3aac06da40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.009+0000 7f3ab2c8c640 1 --2- 172.21.15.16:0/793375010 >> v2:172.21.15.156:3300/0 conn(0x7f3aac144b20 0x7f3aac144f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.009+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/793375010 >> 172.21.15.16:0/793375010 conn(0x7f3aac0b2000 msgr2=0x7f3aac0cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:25.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.009+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/793375010 shutdown_connections 2024-04-24T23:49:25.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.009+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/793375010 wait complete. 2024-04-24T23:49:25.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.010+0000 7f3ab2c8c640 1 Processor -- start 2024-04-24T23:49:25.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.010+0000 7f3ab2c8c640 1 -- start start 2024-04-24T23:49:25.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab2c8c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac0ced90 0x7f3aac143840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab2c8c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aac144b20 0x7f3aac0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab2c8c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac1457b0 0x7f3aac0c9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab2c8c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac0b3390 con 0x7f3aac0ced90 2024-04-24T23:49:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab2c8c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac06d4b0 con 0x7f3aac144b20 2024-04-24T23:49:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab2c8c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f3aac15f990 con 0x7f3aac1457b0 2024-04-24T23:49:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab0a28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac0ced90 0x7f3aac143840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab0a28640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac0ced90 0x7f3aac143840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48720/0 (socket says 172.21.15.16:48720) 2024-04-24T23:49:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab0a28640 1 -- 172.21.15.16:0/1066192321 learned_addr learned my addr 172.21.15.16:0/1066192321 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3aabfff640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aac144b20 0x7f3aac0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab1229640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac1457b0 0x7f3aac0c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab0a28640 1 -- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac1457b0 msgr2=0x7f3aac0c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab0a28640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac1457b0 0x7f3aac0c9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab0a28640 1 -- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aac144b20 msgr2=0x7f3aac0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab0a28640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aac144b20 0x7f3aac0cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab0a28640 1 -- 172.21.15.16:0/1066192321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aac0cab10 con 0x7f3aac0ced90 2024-04-24T23:49:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab1229640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac1457b0 0x7f3aac0c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab0a28640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac0ced90 0x7f3aac143840 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000d800 tx=0x7f3aa000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3aa97fa640 1 -- 172.21.15.16:0/1066192321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa000b6c0 con 0x7f3aac0ced90 2024-04-24T23:49:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3aa97fa640 1 -- 172.21.15.16:0/1066192321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3aa0019070 con 0x7f3aac0ced90 2024-04-24T23:49:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/1066192321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aac0cadb0 con 0x7f3aac0ced90 2024-04-24T23:49:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.011+0000 7f3aa97fa640 1 -- 172.21.15.16:0/1066192321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa000ba80 con 0x7f3aac0ced90 2024-04-24T23:49:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.012+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/1066192321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aac144100 con 0x7f3aac0ced90 2024-04-24T23:49:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.013+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/1066192321 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aac1442d0 con 0x7f3aac0ced90 2024-04-24T23:49:25.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.013+0000 7f3aa97fa640 1 -- 172.21.15.16:0/1066192321 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3aa000bc20 con 0x7f3aac0ced90 2024-04-24T23:49:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.013+0000 7f3aa97fa640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a9006be30 0x7f3a9006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.014+0000 7f3aa97fa640 1 -- 172.21.15.16:0/1066192321 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f3aa008b660 con 0x7f3aac0ced90 2024-04-24T23:49:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.014+0000 7f3aabfff640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a9006be30 0x7f3a9006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.014+0000 7f3aabfff640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a9006be30 0x7f3a9006e2f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3aac1408c0 tx=0x7f3a98009060 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:25.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.018+0000 7f3aa97fa640 1 -- 172.21.15.16:0/1066192321 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3aa0059c30 con 0x7f3aac0ced90 2024-04-24T23:49:25.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.198+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/1066192321 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3aac15bec0 con 0x7f3a9006be30 2024-04-24T23:49:25.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.207+0000 7f3aa97fa640 1 -- 172.21.15.16:0/1066192321 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332531 (secure 0 0 0) 0x7f3aac15bec0 con 0x7f3a9006be30 2024-04-24T23:49:25.210 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:25.211 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-24T23:49:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.210+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a9006be30 msgr2=0x7f3a9006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.210+0000 7f3ab2c8c640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a9006be30 0x7f3a9006e2f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3aac1408c0 tx=0x7f3a98009060 comp rx=0 tx=0).stop 2024-04-24T23:49:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.210+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/1066192321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac0ced90 msgr2=0x7f3aac143840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.210+0000 7f3ab2c8c640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac0ced90 0x7f3aac143840 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3aa000d800 tx=0x7f3aa000dcc0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.210+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/1066192321 shutdown_connections 2024-04-24T23:49:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.210+0000 7f3ab2c8c640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a9006be30 0x7f3a9006e2f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.210+0000 7f3ab2c8c640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac1457b0 0x7f3aac0c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.210+0000 7f3ab2c8c640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aac144b20 0x7f3aac0cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.210+0000 7f3ab2c8c640 1 --2- 172.21.15.16:0/1066192321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac0ced90 0x7f3aac143840 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.210+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/1066192321 >> 172.21.15.16:0/1066192321 conn(0x7f3aac0b2000 msgr2=0x7f3aac070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:25.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.210+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/1066192321 shutdown_connections 2024-04-24T23:49:25.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.210+0000 7f3ab2c8c640 1 -- 172.21.15.16:0/1066192321 wait complete. 2024-04-24T23:49:25.221 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-04-24T23:49:25.124160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248620,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754726100,"statfs":{"total":773094113280,"available":772839526400,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003010"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864269+0000","last_change":"2024-04-24T23:48:54.496506+0000","last_active":"2024-04-24T23:49:00.864269+0000","last_peered":"2024-04-24T23:49:00.864269+0000","last_clean":"2024-04-24T23:49:00.864269+0000","last_became_active":"2024-04-24T23:48:54.496184+0000","last_became_peered":"2024-04-24T23:48:54.496184+0000","last_unstale":"2024-04-24T23:49:00.864269+0000","last_undegraded":"2024-04-24T23:49:00.864269+0000","last_fullsized":"2024-04-24T23:49:00.864269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:26.511856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526502+0000","last_change":"2024-04-24T23:48:55.493918+0000","last_active":"2024-04-24T23:49:00.526502+0000","last_peered":"2024-04-24T23:49:00.526502+0000","last_clean":"2024-04-24T23:49:00.526502+0000","last_became_active":"2024-04-24T23:48:55.493534+0000","last_became_peered":"2024-04-24T23:48:55.493534+0000","last_unstale":"2024-04-24T23:49:00.526502+0000","last_undegraded":"2024-04-24T23:49:00.526502+0000","last_fullsized":"2024-04-24T23:49:00.526502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:56:24.335063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.525963+0000","last_change":"2024-04-24T23:48:59.535521+0000","last_active":"2024-04-24T23:49:00.525963+0000","last_peered":"2024-04-24T23:49:00.525963+0000","last_clean":"2024-04-24T23:49:00.525963+0000","last_became_active":"2024-04-24T23:48:59.535120+0000","last_became_peered":"2024-04-24T23:48:59.535120+0000","last_unstale":"2024-04-24T23:49:00.525963+0000","last_undegraded":"2024-04-24T23:49:00.525963+0000","last_fullsized":"2024-04-24T23:49:00.525963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:02:34.201356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526615+0000","last_change":"2024-04-24T23:48:54.494612+0000","last_active":"2024-04-24T23:49:00.526615+0000","last_peered":"2024-04-24T23:49:00.526615+0000","last_clean":"2024-04-24T23:49:00.526615+0000","last_became_active":"2024-04-24T23:48:54.494317+0000","last_became_peered":"2024-04-24T23:48:54.494317+0000","last_unstale":"2024-04-24T23:49:00.526615+0000","last_undegraded":"2024-04-24T23:49:00.526615+0000","last_fullsized":"2024-04-24T23:49:00.526615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:07.349136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864094+0000","last_change":"2024-04-24T23:48:55.512152+0000","last_active":"2024-04-24T23:49:00.864094+0000","last_peered":"2024-04-24T23:49:00.864094+0000","last_clean":"2024-04-24T23:49:00.864094+0000","last_became_active":"2024-04-24T23:48:55.511396+0000","last_became_peered":"2024-04-24T23:48:55.511396+0000","last_unstale":"2024-04-24T23:49:00.864094+0000","last_undegraded":"2024-04-24T23:49:00.864094+0000","last_fullsized":"2024-04-24T23:49:00.864094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510457+0000","last_change":"2024-04-24T23:48:59.530584+0000","last_active":"2024-04-24T23:49:00.510457+0000","last_peered":"2024-04-24T23:49:00.510457+0000","last_clean":"2024-04-24T23:49:00.510457+0000","last_became_active":"2024-04-24T23:48:59.530081+0000","last_became_peered":"2024-04-24T23:48:59.530081+0000","last_unstale":"2024-04-24T23:49:00.510457+0000","last_undegraded":"2024-04-24T23:49:00.510457+0000","last_fullsized":"2024-04-24T23:49:00.510457+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:03:32.497072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510851+0000","last_change":"2024-04-24T23:48:54.495374+0000","last_active":"2024-04-24T23:49:00.510851+0000","last_peered":"2024-04-24T23:49:00.510851+0000","last_clean":"2024-04-24T23:49:00.510851+0000","last_became_active":"2024-04-24T23:48:54.495040+0000","last_became_peered":"2024-04-24T23:48:54.495040+0000","last_unstale":"2024-04-24T23:49:00.510851+0000","last_undegraded":"2024-04-24T23:49:00.510851+0000","last_fullsized":"2024-04-24T23:49:00.510851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:31.412349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864681+0000","last_change":"2024-04-24T23:48:55.505099+0000","last_active":"2024-04-24T23:49:00.864681+0000","last_peered":"2024-04-24T23:49:00.864681+0000","last_clean":"2024-04-24T23:49:00.864681+0000","last_became_active":"2024-04-24T23:48:55.504778+0000","last_became_peered":"2024-04-24T23:48:55.504778+0000","last_unstale":"2024-04-24T23:49:00.864681+0000","last_undegraded":"2024-04-24T23:49:00.864681+0000","last_fullsized":"2024-04-24T23:49:00.864681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:17.488901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511291+0000","last_change":"2024-04-24T23:48:59.526537+0000","last_active":"2024-04-24T23:49:00.511291+0000","last_peered":"2024-04-24T23:49:00.511291+0000","last_clean":"2024-04-24T23:49:00.511291+0000","last_became_active":"2024-04-24T23:48:59.526038+0000","last_became_peered":"2024-04-24T23:48:59.526038+0000","last_unstale":"2024-04-24T23:49:00.511291+0000","last_undegraded":"2024-04-24T23:49:00.511291+0000","last_fullsized":"2024-04-24T23:49:00.511291+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510029+0000","last_change":"2024-04-24T23:48:54.498311+0000","last_active":"2024-04-24T23:49:00.510029+0000","last_peered":"2024-04-24T23:49:00.510029+0000","last_clean":"2024-04-24T23:49:00.510029+0000","last_became_active":"2024-04-24T23:48:54.497986+0000","last_became_peered":"2024-04-24T23:48:54.497986+0000","last_unstale":"2024-04-24T23:49:00.510029+0000","last_undegraded":"2024-04-24T23:49:00.510029+0000","last_fullsized":"2024-04-24T23:49:00.510029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865047+0000","last_change":"2024-04-24T23:48:55.507029+0000","last_active":"2024-04-24T23:49:00.865047+0000","last_peered":"2024-04-24T23:49:00.865047+0000","last_clean":"2024-04-24T23:49:00.865047+0000","last_became_active":"2024-04-24T23:48:55.506772+0000","last_became_peered":"2024-04-24T23:48:55.506772+0000","last_unstale":"2024-04-24T23:49:00.865047+0000","last_undegraded":"2024-04-24T23:49:00.865047+0000","last_fullsized":"2024-04-24T23:49:00.865047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:42.483054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509920+0000","last_change":"2024-04-24T23:48:59.530333+0000","last_active":"2024-04-24T23:49:00.509920+0000","last_peered":"2024-04-24T23:49:00.509920+0000","last_clean":"2024-04-24T23:49:00.509920+0000","last_became_active":"2024-04-24T23:48:59.529831+0000","last_became_peered":"2024-04-24T23:48:59.529831+0000","last_unstale":"2024-04-24T23:49:00.509920+0000","last_undegraded":"2024-04-24T23:49:00.509920+0000","last_fullsized":"2024-04-24T23:49:00.509920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:47.852102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864057+0000","last_change":"2024-04-24T23:48:55.504704+0000","last_active":"2024-04-24T23:49:00.864057+0000","last_peered":"2024-04-24T23:49:00.864057+0000","last_clean":"2024-04-24T23:49:00.864057+0000","last_became_active":"2024-04-24T23:48:55.504444+0000","last_became_peered":"2024-04-24T23:48:55.504444+0000","last_unstale":"2024-04-24T23:49:00.864057+0000","last_undegraded":"2024-04-24T23:49:00.864057+0000","last_fullsized":"2024-04-24T23:49:00.864057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510280+0000","last_change":"2024-04-24T23:48:54.493270+0000","last_active":"2024-04-24T23:49:00.510280+0000","last_peered":"2024-04-24T23:49:00.510280+0000","last_clean":"2024-04-24T23:49:00.510280+0000","last_became_active":"2024-04-24T23:48:54.492952+0000","last_became_peered":"2024-04-24T23:48:54.492952+0000","last_unstale":"2024-04-24T23:49:00.510280+0000","last_undegraded":"2024-04-24T23:49:00.510280+0000","last_fullsized":"2024-04-24T23:49:00.510280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:36:57.277988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510165+0000","last_change":"2024-04-24T23:48:59.530237+0000","last_active":"2024-04-24T23:49:00.510165+0000","last_peered":"2024-04-24T23:49:00.510165+0000","last_clean":"2024-04-24T23:49:00.510165+0000","last_became_active":"2024-04-24T23:48:59.529732+0000","last_became_peered":"2024-04-24T23:48:59.529732+0000","last_unstale":"2024-04-24T23:49:00.510165+0000","last_undegraded":"2024-04-24T23:49:00.510165+0000","last_fullsized":"2024-04-24T23:49:00.510165+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:59.783695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526172+0000","last_change":"2024-04-24T23:48:55.500061+0000","last_active":"2024-04-24T23:49:00.526172+0000","last_peered":"2024-04-24T23:49:00.526172+0000","last_clean":"2024-04-24T23:49:00.526172+0000","last_became_active":"2024-04-24T23:48:55.499515+0000","last_became_peered":"2024-04-24T23:48:55.499515+0000","last_unstale":"2024-04-24T23:49:00.526172+0000","last_undegraded":"2024-04-24T23:49:00.526172+0000","last_fullsized":"2024-04-24T23:49:00.526172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:10.235664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509506+0000","last_change":"2024-04-24T23:48:54.496643+0000","last_active":"2024-04-24T23:49:00.509506+0000","last_peered":"2024-04-24T23:49:00.509506+0000","last_clean":"2024-04-24T23:49:00.509506+0000","last_became_active":"2024-04-24T23:48:54.496367+0000","last_became_peered":"2024-04-24T23:48:54.496367+0000","last_unstale":"2024-04-24T23:49:00.509506+0000","last_undegraded":"2024-04-24T23:49:00.509506+0000","last_fullsized":"2024-04-24T23:49:00.509506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:17.375124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510348+0000","last_change":"2024-04-24T23:48:59.531327+0000","last_active":"2024-04-24T23:49:00.510348+0000","last_peered":"2024-04-24T23:49:00.510348+0000","last_clean":"2024-04-24T23:49:00.510348+0000","last_became_active":"2024-04-24T23:48:59.530959+0000","last_became_peered":"2024-04-24T23:48:59.530959+0000","last_unstale":"2024-04-24T23:49:00.510348+0000","last_undegraded":"2024-04-24T23:49:00.510348+0000","last_fullsized":"2024-04-24T23:49:00.510348+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:14.260616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526263+0000","last_change":"2024-04-24T23:48:55.494442+0000","last_active":"2024-04-24T23:49:00.526263+0000","last_peered":"2024-04-24T23:49:00.526263+0000","last_clean":"2024-04-24T23:49:00.526263+0000","last_became_active":"2024-04-24T23:48:55.494165+0000","last_became_peered":"2024-04-24T23:48:55.494165+0000","last_unstale":"2024-04-24T23:49:00.526263+0000","last_undegraded":"2024-04-24T23:49:00.526263+0000","last_fullsized":"2024-04-24T23:49:00.526263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508952+0000","last_change":"2024-04-24T23:48:54.488761+0000","last_active":"2024-04-24T23:49:00.508952+0000","last_peered":"2024-04-24T23:49:00.508952+0000","last_clean":"2024-04-24T23:49:00.508952+0000","last_became_active":"2024-04-24T23:48:54.488431+0000","last_became_peered":"2024-04-24T23:48:54.488431+0000","last_unstale":"2024-04-24T23:49:00.508952+0000","last_undegraded":"2024-04-24T23:49:00.508952+0000","last_fullsized":"2024-04-24T23:49:00.508952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509720+0000","last_change":"2024-04-24T23:48:59.523973+0000","last_active":"2024-04-24T23:49:00.509720+0000","last_peered":"2024-04-24T23:49:00.509720+0000","last_clean":"2024-04-24T23:49:00.509720+0000","last_became_active":"2024-04-24T23:48:59.520679+0000","last_became_peered":"2024-04-24T23:48:59.520679+0000","last_unstale":"2024-04-24T23:49:00.509720+0000","last_undegraded":"2024-04-24T23:49:00.509720+0000","last_fullsized":"2024-04-24T23:49:00.509720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511072+0000","last_change":"2024-04-24T23:48:55.505618+0000","last_active":"2024-04-24T23:49:00.511072+0000","last_peered":"2024-04-24T23:49:00.511072+0000","last_clean":"2024-04-24T23:49:00.511072+0000","last_became_active":"2024-04-24T23:48:55.504804+0000","last_became_peered":"2024-04-24T23:48:55.504804+0000","last_unstale":"2024-04-24T23:49:00.511072+0000","last_undegraded":"2024-04-24T23:49:00.511072+0000","last_fullsized":"2024-04-24T23:49:00.511072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:46.215368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509409+0000","last_change":"2024-04-24T23:48:54.496898+0000","last_active":"2024-04-24T23:49:00.509409+0000","last_peered":"2024-04-24T23:49:00.509409+0000","last_clean":"2024-04-24T23:49:00.509409+0000","last_became_active":"2024-04-24T23:48:54.496603+0000","last_became_peered":"2024-04-24T23:48:54.496603+0000","last_unstale":"2024-04-24T23:49:00.509409+0000","last_undegraded":"2024-04-24T23:49:00.509409+0000","last_fullsized":"2024-04-24T23:49:00.509409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510695+0000","last_change":"2024-04-24T23:48:59.533868+0000","last_active":"2024-04-24T23:49:00.510695+0000","last_peered":"2024-04-24T23:49:00.510695+0000","last_clean":"2024-04-24T23:49:00.510695+0000","last_became_active":"2024-04-24T23:48:59.533487+0000","last_became_peered":"2024-04-24T23:48:59.533487+0000","last_unstale":"2024-04-24T23:49:00.510695+0000","last_undegraded":"2024-04-24T23:49:00.510695+0000","last_fullsized":"2024-04-24T23:49:00.510695+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:21.513853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864071+0000","last_change":"2024-04-24T23:48:55.497734+0000","last_active":"2024-04-24T23:49:00.864071+0000","last_peered":"2024-04-24T23:49:00.864071+0000","last_clean":"2024-04-24T23:49:00.864071+0000","last_became_active":"2024-04-24T23:48:55.496634+0000","last_became_peered":"2024-04-24T23:48:55.496634+0000","last_unstale":"2024-04-24T23:49:00.864071+0000","last_undegraded":"2024-04-24T23:49:00.864071+0000","last_fullsized":"2024-04-24T23:49:00.864071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:56:24.335063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864834+0000","last_change":"2024-04-24T23:48:54.499625+0000","last_active":"2024-04-24T23:49:00.864834+0000","last_peered":"2024-04-24T23:49:00.864834+0000","last_clean":"2024-04-24T23:49:00.864834+0000","last_became_active":"2024-04-24T23:48:54.499357+0000","last_became_peered":"2024-04-24T23:48:54.499357+0000","last_unstale":"2024-04-24T23:49:00.864834+0000","last_undegraded":"2024-04-24T23:49:00.864834+0000","last_fullsized":"2024-04-24T23:49:00.864834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511165+0000","last_change":"2024-04-24T23:48:59.523817+0000","last_active":"2024-04-24T23:49:00.511165+0000","last_peered":"2024-04-24T23:49:00.511165+0000","last_clean":"2024-04-24T23:49:00.511165+0000","last_became_active":"2024-04-24T23:48:59.522505+0000","last_became_peered":"2024-04-24T23:48:59.522505+0000","last_unstale":"2024-04-24T23:49:00.511165+0000","last_undegraded":"2024-04-24T23:49:00.511165+0000","last_fullsized":"2024-04-24T23:49:00.511165+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:37.247560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526836+0000","last_change":"2024-04-24T23:48:55.490605+0000","last_active":"2024-04-24T23:49:00.526836+0000","last_peered":"2024-04-24T23:49:00.526836+0000","last_clean":"2024-04-24T23:49:00.526836+0000","last_became_active":"2024-04-24T23:48:55.490271+0000","last_became_peered":"2024-04-24T23:48:55.490271+0000","last_unstale":"2024-04-24T23:49:00.526836+0000","last_undegraded":"2024-04-24T23:49:00.526836+0000","last_fullsized":"2024-04-24T23:49:00.526836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:53:38.263114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509803+0000","last_change":"2024-04-24T23:48:54.493101+0000","last_active":"2024-04-24T23:49:00.509803+0000","last_peered":"2024-04-24T23:49:00.509803+0000","last_clean":"2024-04-24T23:49:00.509803+0000","last_became_active":"2024-04-24T23:48:54.492768+0000","last_became_peered":"2024-04-24T23:48:54.492768+0000","last_unstale":"2024-04-24T23:49:00.509803+0000","last_undegraded":"2024-04-24T23:49:00.509803+0000","last_fullsized":"2024-04-24T23:49:00.509803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:15:10.269984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.525949+0000","last_change":"2024-04-24T23:48:59.535686+0000","last_active":"2024-04-24T23:49:00.525949+0000","last_peered":"2024-04-24T23:49:00.525949+0000","last_clean":"2024-04-24T23:49:00.525949+0000","last_became_active":"2024-04-24T23:48:59.535320+0000","last_became_peered":"2024-04-24T23:48:59.535320+0000","last_unstale":"2024-04-24T23:49:00.525949+0000","last_undegraded":"2024-04-24T23:49:00.525949+0000","last_fullsized":"2024-04-24T23:49:00.525949+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:22.709044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509618+0000","last_change":"2024-04-24T23:48:55.493935+0000","last_active":"2024-04-24T23:49:00.509618+0000","last_peered":"2024-04-24T23:49:00.509618+0000","last_clean":"2024-04-24T23:49:00.509618+0000","last_became_active":"2024-04-24T23:48:55.493604+0000","last_became_peered":"2024-04-24T23:48:55.493604+0000","last_unstale":"2024-04-24T23:49:00.509618+0000","last_undegraded":"2024-04-24T23:49:00.509618+0000","last_fullsized":"2024-04-24T23:49:00.509618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510971+0000","last_change":"2024-04-24T23:48:54.497633+0000","last_active":"2024-04-24T23:49:00.510971+0000","last_peered":"2024-04-24T23:49:00.510971+0000","last_clean":"2024-04-24T23:49:00.510971+0000","last_became_active":"2024-04-24T23:48:54.497329+0000","last_became_peered":"2024-04-24T23:48:54.497329+0000","last_unstale":"2024-04-24T23:49:00.510971+0000","last_undegraded":"2024-04-24T23:49:00.510971+0000","last_fullsized":"2024-04-24T23:49:00.510971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:37.806706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864823+0000","last_change":"2024-04-24T23:48:59.538720+0000","last_active":"2024-04-24T23:49:00.864823+0000","last_peered":"2024-04-24T23:49:00.864823+0000","last_clean":"2024-04-24T23:49:00.864823+0000","last_became_active":"2024-04-24T23:48:59.538310+0000","last_became_peered":"2024-04-24T23:48:59.538310+0000","last_unstale":"2024-04-24T23:49:00.864823+0000","last_undegraded":"2024-04-24T23:49:00.864823+0000","last_fullsized":"2024-04-24T23:49:00.864823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:10.292844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865110+0000","last_change":"2024-04-24T23:48:55.510066+0000","last_active":"2024-04-24T23:49:00.865110+0000","last_peered":"2024-04-24T23:49:00.865110+0000","last_clean":"2024-04-24T23:49:00.865110+0000","last_became_active":"2024-04-24T23:48:55.507977+0000","last_became_peered":"2024-04-24T23:48:55.507977+0000","last_unstale":"2024-04-24T23:49:00.865110+0000","last_undegraded":"2024-04-24T23:49:00.865110+0000","last_fullsized":"2024-04-24T23:49:00.865110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:06.267893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509767+0000","last_change":"2024-04-24T23:48:54.493032+0000","last_active":"2024-04-24T23:49:00.509767+0000","last_peered":"2024-04-24T23:49:00.509767+0000","last_clean":"2024-04-24T23:49:00.509767+0000","last_became_active":"2024-04-24T23:48:54.492694+0000","last_became_peered":"2024-04-24T23:48:54.492694+0000","last_unstale":"2024-04-24T23:49:00.509767+0000","last_undegraded":"2024-04-24T23:49:00.509767+0000","last_fullsized":"2024-04-24T23:49:00.509767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509213+0000","last_change":"2024-04-24T23:48:59.527964+0000","last_active":"2024-04-24T23:49:00.509213+0000","last_peered":"2024-04-24T23:49:00.509213+0000","last_clean":"2024-04-24T23:49:00.509213+0000","last_became_active":"2024-04-24T23:48:59.527519+0000","last_became_peered":"2024-04-24T23:48:59.527519+0000","last_unstale":"2024-04-24T23:49:00.509213+0000","last_undegraded":"2024-04-24T23:49:00.509213+0000","last_fullsized":"2024-04-24T23:49:00.509213+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:37.247560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511053+0000","last_change":"2024-04-24T23:48:55.505703+0000","last_active":"2024-04-24T23:49:00.511053+0000","last_peered":"2024-04-24T23:49:00.511053+0000","last_clean":"2024-04-24T23:49:00.511053+0000","last_became_active":"2024-04-24T23:48:55.505257+0000","last_became_peered":"2024-04-24T23:48:55.505257+0000","last_unstale":"2024-04-24T23:49:00.511053+0000","last_undegraded":"2024-04-24T23:49:00.511053+0000","last_fullsized":"2024-04-24T23:49:00.511053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:41.372867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864734+0000","last_change":"2024-04-24T23:48:54.494961+0000","last_active":"2024-04-24T23:49:00.864734+0000","last_peered":"2024-04-24T23:49:00.864734+0000","last_clean":"2024-04-24T23:49:00.864734+0000","last_became_active":"2024-04-24T23:48:54.494672+0000","last_became_peered":"2024-04-24T23:48:54.494672+0000","last_unstale":"2024-04-24T23:49:00.864734+0000","last_undegraded":"2024-04-24T23:49:00.864734+0000","last_fullsized":"2024-04-24T23:49:00.864734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509770+0000","last_change":"2024-04-24T23:48:59.525981+0000","last_active":"2024-04-24T23:49:00.509770+0000","last_peered":"2024-04-24T23:49:00.509770+0000","last_clean":"2024-04-24T23:49:00.509770+0000","last_became_active":"2024-04-24T23:48:59.524788+0000","last_became_peered":"2024-04-24T23:48:59.524788+0000","last_unstale":"2024-04-24T23:49:00.509770+0000","last_undegraded":"2024-04-24T23:49:00.509770+0000","last_fullsized":"2024-04-24T23:49:00.509770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:08.826500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508934+0000","last_change":"2024-04-24T23:48:55.493966+0000","last_active":"2024-04-24T23:49:00.508934+0000","last_peered":"2024-04-24T23:49:00.508934+0000","last_clean":"2024-04-24T23:49:00.508934+0000","last_became_active":"2024-04-24T23:48:55.493622+0000","last_became_peered":"2024-04-24T23:48:55.493622+0000","last_unstale":"2024-04-24T23:49:00.508934+0000","last_undegraded":"2024-04-24T23:49:00.508934+0000","last_fullsized":"2024-04-24T23:49:00.508934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863565+0000","last_change":"2024-04-24T23:48:54.497811+0000","last_active":"2024-04-24T23:49:00.863565+0000","last_peered":"2024-04-24T23:49:00.863565+0000","last_clean":"2024-04-24T23:49:00.863565+0000","last_became_active":"2024-04-24T23:48:54.497465+0000","last_became_peered":"2024-04-24T23:48:54.497465+0000","last_unstale":"2024-04-24T23:49:00.863565+0000","last_undegraded":"2024-04-24T23:49:00.863565+0000","last_fullsized":"2024-04-24T23:49:00.863565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:17.375124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509953+0000","last_change":"2024-04-24T23:48:59.536596+0000","last_active":"2024-04-24T23:49:00.509953+0000","last_peered":"2024-04-24T23:49:00.509953+0000","last_clean":"2024-04-24T23:49:00.509953+0000","last_became_active":"2024-04-24T23:48:59.536186+0000","last_became_peered":"2024-04-24T23:48:59.536186+0000","last_unstale":"2024-04-24T23:49:00.509953+0000","last_undegraded":"2024-04-24T23:49:00.509953+0000","last_fullsized":"2024-04-24T23:49:00.509953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:46.508006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863695+0000","last_change":"2024-04-24T23:48:55.507490+0000","last_active":"2024-04-24T23:49:00.863695+0000","last_peered":"2024-04-24T23:49:00.863695+0000","last_clean":"2024-04-24T23:49:00.863695+0000","last_became_active":"2024-04-24T23:48:55.506415+0000","last_became_peered":"2024-04-24T23:48:55.506415+0000","last_unstale":"2024-04-24T23:49:00.863695+0000","last_undegraded":"2024-04-24T23:49:00.863695+0000","last_fullsized":"2024-04-24T23:49:00.863695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509313+0000","last_change":"2024-04-24T23:48:54.491388+0000","last_active":"2024-04-24T23:49:00.509313+0000","last_peered":"2024-04-24T23:49:00.509313+0000","last_clean":"2024-04-24T23:49:00.509313+0000","last_became_active":"2024-04-24T23:48:54.490991+0000","last_became_peered":"2024-04-24T23:48:54.490991+0000","last_unstale":"2024-04-24T23:49:00.509313+0000","last_undegraded":"2024-04-24T23:49:00.509313+0000","last_fullsized":"2024-04-24T23:49:00.509313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863944+0000","last_change":"2024-04-24T23:48:59.539922+0000","last_active":"2024-04-24T23:49:00.863944+0000","last_peered":"2024-04-24T23:49:00.863944+0000","last_clean":"2024-04-24T23:49:00.863944+0000","last_became_active":"2024-04-24T23:48:59.539449+0000","last_became_peered":"2024-04-24T23:48:59.539449+0000","last_unstale":"2024-04-24T23:49:00.863944+0000","last_undegraded":"2024-04-24T23:49:00.863944+0000","last_fullsized":"2024-04-24T23:49:00.863944+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:08.826500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526078+0000","last_change":"2024-04-24T23:48:55.509237+0000","last_active":"2024-04-24T23:49:00.526078+0000","last_peered":"2024-04-24T23:49:00.526078+0000","last_clean":"2024-04-24T23:49:00.526078+0000","last_became_active":"2024-04-24T23:48:55.508847+0000","last_became_peered":"2024-04-24T23:48:55.508847+0000","last_unstale":"2024-04-24T23:49:00.526078+0000","last_undegraded":"2024-04-24T23:49:00.526078+0000","last_fullsized":"2024-04-24T23:49:00.526078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864955+0000","last_change":"2024-04-24T23:48:54.496310+0000","last_active":"2024-04-24T23:49:00.864955+0000","last_peered":"2024-04-24T23:49:00.864955+0000","last_clean":"2024-04-24T23:49:00.864955+0000","last_became_active":"2024-04-24T23:48:54.496016+0000","last_became_peered":"2024-04-24T23:48:54.496016+0000","last_unstale":"2024-04-24T23:49:00.864955+0000","last_undegraded":"2024-04-24T23:49:00.864955+0000","last_fullsized":"2024-04-24T23:49:00.864955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509522+0000","last_change":"2024-04-24T23:48:45.082459+0000","last_active":"2024-04-24T23:49:00.509522+0000","last_peered":"2024-04-24T23:49:00.509522+0000","last_clean":"2024-04-24T23:49:00.509522+0000","last_became_active":"2024-04-24T23:48:45.081495+0000","last_became_peered":"2024-04-24T23:48:45.081495+0000","last_unstale":"2024-04-24T23:49:00.509522+0000","last_undegraded":"2024-04-24T23:49:00.509522+0000","last_fullsized":"2024-04-24T23:49:00.509522+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_clean_scrub_stamp":"2024-04-24T23:48:43.440993+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:53:20.832511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864431+0000","last_change":"2024-04-24T23:48:59.521532+0000","last_active":"2024-04-24T23:49:00.864431+0000","last_peered":"2024-04-24T23:49:00.864431+0000","last_clean":"2024-04-24T23:49:00.864431+0000","last_became_active":"2024-04-24T23:48:59.521144+0000","last_became_peered":"2024-04-24T23:48:59.521144+0000","last_unstale":"2024-04-24T23:49:00.864431+0000","last_undegraded":"2024-04-24T23:49:00.864431+0000","last_fullsized":"2024-04-24T23:49:00.864431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:08.826500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510601+0000","last_change":"2024-04-24T23:48:55.509123+0000","last_active":"2024-04-24T23:49:00.510601+0000","last_peered":"2024-04-24T23:49:00.510601+0000","last_clean":"2024-04-24T23:49:00.510601+0000","last_became_active":"2024-04-24T23:48:55.508860+0000","last_became_peered":"2024-04-24T23:48:55.508860+0000","last_unstale":"2024-04-24T23:49:00.510601+0000","last_undegraded":"2024-04-24T23:49:00.510601+0000","last_fullsized":"2024-04-24T23:49:00.510601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:03:28.472120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863884+0000","last_change":"2024-04-24T23:48:54.498741+0000","last_active":"2024-04-24T23:49:00.863884+0000","last_peered":"2024-04-24T23:49:00.863884+0000","last_clean":"2024-04-24T23:49:00.863884+0000","last_became_active":"2024-04-24T23:48:54.498241+0000","last_became_peered":"2024-04-24T23:48:54.498241+0000","last_unstale":"2024-04-24T23:49:00.863884+0000","last_undegraded":"2024-04-24T23:49:00.863884+0000","last_fullsized":"2024-04-24T23:49:00.863884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:07.349136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864020+0000","last_change":"2024-04-24T23:48:59.528434+0000","last_active":"2024-04-24T23:49:00.864020+0000","last_peered":"2024-04-24T23:49:00.864020+0000","last_clean":"2024-04-24T23:49:00.864020+0000","last_became_active":"2024-04-24T23:48:59.528062+0000","last_became_peered":"2024-04-24T23:48:59.528062+0000","last_unstale":"2024-04-24T23:49:00.864020+0000","last_undegraded":"2024-04-24T23:49:00.864020+0000","last_fullsized":"2024-04-24T23:49:00.864020+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864801+0000","last_change":"2024-04-24T23:48:55.509799+0000","last_active":"2024-04-24T23:49:00.864801+0000","last_peered":"2024-04-24T23:49:00.864801+0000","last_clean":"2024-04-24T23:49:00.864801+0000","last_became_active":"2024-04-24T23:48:55.509286+0000","last_became_peered":"2024-04-24T23:48:55.509286+0000","last_unstale":"2024-04-24T23:49:00.864801+0000","last_undegraded":"2024-04-24T23:49:00.864801+0000","last_fullsized":"2024-04-24T23:49:00.864801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:33.222608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509751+0000","last_change":"2024-04-24T23:48:54.490614+0000","last_active":"2024-04-24T23:49:00.509751+0000","last_peered":"2024-04-24T23:49:00.509751+0000","last_clean":"2024-04-24T23:49:00.509751+0000","last_became_active":"2024-04-24T23:48:54.490029+0000","last_became_peered":"2024-04-24T23:48:54.490029+0000","last_unstale":"2024-04-24T23:49:00.509751+0000","last_undegraded":"2024-04-24T23:49:00.509751+0000","last_fullsized":"2024-04-24T23:49:00.509751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864962+0000","last_change":"2024-04-24T23:48:59.529966+0000","last_active":"2024-04-24T23:49:00.864962+0000","last_peered":"2024-04-24T23:49:00.864962+0000","last_clean":"2024-04-24T23:49:00.864962+0000","last_became_active":"2024-04-24T23:48:59.529508+0000","last_became_peered":"2024-04-24T23:48:59.529508+0000","last_unstale":"2024-04-24T23:49:00.864962+0000","last_undegraded":"2024-04-24T23:49:00.864962+0000","last_fullsized":"2024-04-24T23:49:00.864962+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:46.508006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511361+0000","last_change":"2024-04-24T23:48:55.506681+0000","last_active":"2024-04-24T23:49:00.511361+0000","last_peered":"2024-04-24T23:49:00.511361+0000","last_clean":"2024-04-24T23:49:00.511361+0000","last_became_active":"2024-04-24T23:48:55.506393+0000","last_became_peered":"2024-04-24T23:48:55.506393+0000","last_unstale":"2024-04-24T23:49:00.511361+0000","last_undegraded":"2024-04-24T23:49:00.511361+0000","last_fullsized":"2024-04-24T23:49:00.511361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:02:30.176405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863990+0000","last_change":"2024-04-24T23:48:54.499321+0000","last_active":"2024-04-24T23:49:00.863990+0000","last_peered":"2024-04-24T23:49:00.863990+0000","last_clean":"2024-04-24T23:49:00.863990+0000","last_became_active":"2024-04-24T23:48:54.499015+0000","last_became_peered":"2024-04-24T23:48:54.499015+0000","last_unstale":"2024-04-24T23:49:00.863990+0000","last_undegraded":"2024-04-24T23:49:00.863990+0000","last_fullsized":"2024-04-24T23:49:00.863990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863856+0000","last_change":"2024-04-24T23:48:59.522405+0000","last_active":"2024-04-24T23:49:00.863856+0000","last_peered":"2024-04-24T23:49:00.863856+0000","last_clean":"2024-04-24T23:49:00.863856+0000","last_became_active":"2024-04-24T23:48:59.521434+0000","last_became_peered":"2024-04-24T23:48:59.521434+0000","last_unstale":"2024-04-24T23:49:00.863856+0000","last_undegraded":"2024-04-24T23:49:00.863856+0000","last_fullsized":"2024-04-24T23:49:00.863856+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:39.512436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511297+0000","last_change":"2024-04-24T23:48:55.507123+0000","last_active":"2024-04-24T23:49:00.511297+0000","last_peered":"2024-04-24T23:49:00.511297+0000","last_clean":"2024-04-24T23:49:00.511297+0000","last_became_active":"2024-04-24T23:48:55.506838+0000","last_became_peered":"2024-04-24T23:48:55.506838+0000","last_unstale":"2024-04-24T23:49:00.511297+0000","last_undegraded":"2024-04-24T23:49:00.511297+0000","last_fullsized":"2024-04-24T23:49:00.511297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:17.488901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509852+0000","last_change":"2024-04-24T23:48:54.494375+0000","last_active":"2024-04-24T23:49:00.509852+0000","last_peered":"2024-04-24T23:49:00.509852+0000","last_clean":"2024-04-24T23:49:00.509852+0000","last_became_active":"2024-04-24T23:48:54.494080+0000","last_became_peered":"2024-04-24T23:48:54.494080+0000","last_unstale":"2024-04-24T23:49:00.509852+0000","last_undegraded":"2024-04-24T23:49:00.509852+0000","last_fullsized":"2024-04-24T23:49:00.509852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:46.268530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510042+0000","last_change":"2024-04-24T23:48:59.530805+0000","last_active":"2024-04-24T23:49:00.510042+0000","last_peered":"2024-04-24T23:49:00.510042+0000","last_clean":"2024-04-24T23:49:00.510042+0000","last_became_active":"2024-04-24T23:48:59.530126+0000","last_became_peered":"2024-04-24T23:48:59.530126+0000","last_unstale":"2024-04-24T23:49:00.510042+0000","last_undegraded":"2024-04-24T23:49:00.510042+0000","last_fullsized":"2024-04-24T23:49:00.510042+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:50.914824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509597+0000","last_change":"2024-04-24T23:48:55.499881+0000","last_active":"2024-04-24T23:49:00.509597+0000","last_peered":"2024-04-24T23:49:00.509597+0000","last_clean":"2024-04-24T23:49:00.509597+0000","last_became_active":"2024-04-24T23:48:55.499493+0000","last_became_peered":"2024-04-24T23:48:55.499493+0000","last_unstale":"2024-04-24T23:49:00.509597+0000","last_undegraded":"2024-04-24T23:49:00.509597+0000","last_fullsized":"2024-04-24T23:49:00.509597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:33.222608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863280+0000","last_change":"2024-04-24T23:48:54.493205+0000","last_active":"2024-04-24T23:49:00.863280+0000","last_peered":"2024-04-24T23:49:00.863280+0000","last_clean":"2024-04-24T23:49:00.863280+0000","last_became_active":"2024-04-24T23:48:54.492446+0000","last_became_peered":"2024-04-24T23:48:54.492446+0000","last_unstale":"2024-04-24T23:49:00.863280+0000","last_undegraded":"2024-04-24T23:49:00.863280+0000","last_fullsized":"2024-04-24T23:49:00.863280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:16:25.658826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863935+0000","last_change":"2024-04-24T23:48:59.532383+0000","last_active":"2024-04-24T23:49:00.863935+0000","last_peered":"2024-04-24T23:49:00.863935+0000","last_clean":"2024-04-24T23:49:00.863935+0000","last_became_active":"2024-04-24T23:48:59.529606+0000","last_became_peered":"2024-04-24T23:48:59.529606+0000","last_unstale":"2024-04-24T23:49:00.863935+0000","last_undegraded":"2024-04-24T23:49:00.863935+0000","last_fullsized":"2024-04-24T23:49:00.863935+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864681+0000","last_change":"2024-04-24T23:48:55.499374+0000","last_active":"2024-04-24T23:49:00.864681+0000","last_peered":"2024-04-24T23:49:00.864681+0000","last_clean":"2024-04-24T23:49:00.864681+0000","last_became_active":"2024-04-24T23:48:55.499107+0000","last_became_peered":"2024-04-24T23:48:55.499107+0000","last_unstale":"2024-04-24T23:49:00.864681+0000","last_undegraded":"2024-04-24T23:49:00.864681+0000","last_fullsized":"2024-04-24T23:49:00.864681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864217+0000","last_change":"2024-04-24T23:48:54.491505+0000","last_active":"2024-04-24T23:49:00.864217+0000","last_peered":"2024-04-24T23:49:00.864217+0000","last_clean":"2024-04-24T23:49:00.864217+0000","last_became_active":"2024-04-24T23:48:54.491230+0000","last_became_peered":"2024-04-24T23:48:54.491230+0000","last_unstale":"2024-04-24T23:49:00.864217+0000","last_undegraded":"2024-04-24T23:49:00.864217+0000","last_fullsized":"2024-04-24T23:49:00.864217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526800+0000","last_change":"2024-04-24T23:48:59.531330+0000","last_active":"2024-04-24T23:49:00.526800+0000","last_peered":"2024-04-24T23:49:00.526800+0000","last_clean":"2024-04-24T23:49:00.526800+0000","last_became_active":"2024-04-24T23:48:59.530916+0000","last_became_peered":"2024-04-24T23:48:59.530916+0000","last_unstale":"2024-04-24T23:49:00.526800+0000","last_undegraded":"2024-04-24T23:49:00.526800+0000","last_fullsized":"2024-04-24T23:49:00.526800+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:43.024671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510174+0000","last_change":"2024-04-24T23:48:55.502184+0000","last_active":"2024-04-24T23:49:00.510174+0000","last_peered":"2024-04-24T23:49:00.510174+0000","last_clean":"2024-04-24T23:49:00.510174+0000","last_became_active":"2024-04-24T23:48:55.501682+0000","last_became_peered":"2024-04-24T23:48:55.501682+0000","last_unstale":"2024-04-24T23:49:00.510174+0000","last_undegraded":"2024-04-24T23:49:00.510174+0000","last_fullsized":"2024-04-24T23:49:00.510174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:06.267893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864548+0000","last_change":"2024-04-24T23:48:54.499017+0000","last_active":"2024-04-24T23:49:00.864548+0000","last_peered":"2024-04-24T23:49:00.864548+0000","last_clean":"2024-04-24T23:49:00.864548+0000","last_became_active":"2024-04-24T23:48:54.498747+0000","last_became_peered":"2024-04-24T23:48:54.498747+0000","last_unstale":"2024-04-24T23:49:00.864548+0000","last_undegraded":"2024-04-24T23:49:00.864548+0000","last_fullsized":"2024-04-24T23:49:00.864548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526541+0000","last_change":"2024-04-24T23:48:59.529890+0000","last_active":"2024-04-24T23:49:00.526541+0000","last_peered":"2024-04-24T23:49:00.526541+0000","last_clean":"2024-04-24T23:49:00.526541+0000","last_became_active":"2024-04-24T23:48:59.529534+0000","last_became_peered":"2024-04-24T23:48:59.529534+0000","last_unstale":"2024-04-24T23:49:00.526541+0000","last_undegraded":"2024-04-24T23:49:00.526541+0000","last_fullsized":"2024-04-24T23:49:00.526541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:38:25.945723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864543+0000","last_change":"2024-04-24T23:48:55.497917+0000","last_active":"2024-04-24T23:49:00.864543+0000","last_peered":"2024-04-24T23:49:00.864543+0000","last_clean":"2024-04-24T23:49:00.864543+0000","last_became_active":"2024-04-24T23:48:55.497158+0000","last_became_peered":"2024-04-24T23:48:55.497158+0000","last_unstale":"2024-04-24T23:49:00.864543+0000","last_undegraded":"2024-04-24T23:49:00.864543+0000","last_fullsized":"2024-04-24T23:49:00.864543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:51.930595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865010+0000","last_change":"2024-04-24T23:48:54.491006+0000","last_active":"2024-04-24T23:49:00.865010+0000","last_peered":"2024-04-24T23:49:00.865010+0000","last_clean":"2024-04-24T23:49:00.865010+0000","last_became_active":"2024-04-24T23:48:54.489661+0000","last_became_peered":"2024-04-24T23:48:54.489661+0000","last_unstale":"2024-04-24T23:49:00.865010+0000","last_undegraded":"2024-04-24T23:49:00.865010+0000","last_fullsized":"2024-04-24T23:49:00.865010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:16:25.658826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864966+0000","last_change":"2024-04-24T23:48:59.521225+0000","last_active":"2024-04-24T23:49:00.864966+0000","last_peered":"2024-04-24T23:49:00.864966+0000","last_clean":"2024-04-24T23:49:00.864966+0000","last_became_active":"2024-04-24T23:48:59.520761+0000","last_became_peered":"2024-04-24T23:48:59.520761+0000","last_unstale":"2024-04-24T23:49:00.864966+0000","last_undegraded":"2024-04-24T23:49:00.864966+0000","last_fullsized":"2024-04-24T23:49:00.864966+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:38:25.945723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510500+0000","last_change":"2024-04-24T23:48:55.487564+0000","last_active":"2024-04-24T23:49:00.510500+0000","last_peered":"2024-04-24T23:49:00.510500+0000","last_clean":"2024-04-24T23:49:00.510500+0000","last_became_active":"2024-04-24T23:48:55.487272+0000","last_became_peered":"2024-04-24T23:48:55.487272+0000","last_unstale":"2024-04-24T23:49:00.510500+0000","last_undegraded":"2024-04-24T23:49:00.510500+0000","last_fullsized":"2024-04-24T23:49:00.510500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:10.235664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:04.927990+0000","last_change":"2024-04-24T23:48:54.500818+0000","last_active":"2024-04-24T23:49:04.927990+0000","last_peered":"2024-04-24T23:49:04.927990+0000","last_clean":"2024-04-24T23:49:04.927990+0000","last_became_active":"2024-04-24T23:48:54.500560+0000","last_became_peered":"2024-04-24T23:48:54.500560+0000","last_unstale":"2024-04-24T23:49:04.927990+0000","last_undegraded":"2024-04-24T23:49:04.927990+0000","last_fullsized":"2024-04-24T23:49:04.927990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509881+0000","last_change":"2024-04-24T23:48:59.530292+0000","last_active":"2024-04-24T23:49:00.509881+0000","last_peered":"2024-04-24T23:49:00.509881+0000","last_clean":"2024-04-24T23:49:00.509881+0000","last_became_active":"2024-04-24T23:48:59.529429+0000","last_became_peered":"2024-04-24T23:48:59.529429+0000","last_unstale":"2024-04-24T23:49:00.509881+0000","last_undegraded":"2024-04-24T23:49:00.509881+0000","last_fullsized":"2024-04-24T23:49:00.509881+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:10.292844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510600+0000","last_change":"2024-04-24T23:48:55.502480+0000","last_active":"2024-04-24T23:49:00.510600+0000","last_peered":"2024-04-24T23:49:00.510600+0000","last_clean":"2024-04-24T23:49:00.510600+0000","last_became_active":"2024-04-24T23:48:55.502104+0000","last_became_peered":"2024-04-24T23:48:55.502104+0000","last_unstale":"2024-04-24T23:49:00.510600+0000","last_undegraded":"2024-04-24T23:49:00.510600+0000","last_fullsized":"2024-04-24T23:49:00.510600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:25.555824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509534+0000","last_change":"2024-04-24T23:48:54.495473+0000","last_active":"2024-04-24T23:49:00.509534+0000","last_peered":"2024-04-24T23:49:00.509534+0000","last_clean":"2024-04-24T23:49:00.509534+0000","last_became_active":"2024-04-24T23:48:54.495173+0000","last_became_peered":"2024-04-24T23:48:54.495173+0000","last_unstale":"2024-04-24T23:49:00.509534+0000","last_undegraded":"2024-04-24T23:49:00.509534+0000","last_fullsized":"2024-04-24T23:49:00.509534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510914+0000","last_change":"2024-04-24T23:48:59.532057+0000","last_active":"2024-04-24T23:49:00.510914+0000","last_peered":"2024-04-24T23:49:00.510914+0000","last_clean":"2024-04-24T23:49:00.510914+0000","last_became_active":"2024-04-24T23:48:59.530378+0000","last_became_peered":"2024-04-24T23:48:59.530378+0000","last_unstale":"2024-04-24T23:49:00.510914+0000","last_undegraded":"2024-04-24T23:49:00.510914+0000","last_fullsized":"2024-04-24T23:49:00.510914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:51.652134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864215+0000","last_change":"2024-04-24T23:48:55.508829+0000","last_active":"2024-04-24T23:49:00.864215+0000","last_peered":"2024-04-24T23:49:00.864215+0000","last_clean":"2024-04-24T23:49:00.864215+0000","last_became_active":"2024-04-24T23:48:55.508042+0000","last_became_peered":"2024-04-24T23:48:55.508042+0000","last_unstale":"2024-04-24T23:49:00.864215+0000","last_undegraded":"2024-04-24T23:49:00.864215+0000","last_fullsized":"2024-04-24T23:49:00.864215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865051+0000","last_change":"2024-04-24T23:48:54.497886+0000","last_active":"2024-04-24T23:49:00.865051+0000","last_peered":"2024-04-24T23:49:00.865051+0000","last_clean":"2024-04-24T23:49:00.865051+0000","last_became_active":"2024-04-24T23:48:54.497630+0000","last_became_peered":"2024-04-24T23:48:54.497630+0000","last_unstale":"2024-04-24T23:49:00.865051+0000","last_undegraded":"2024-04-24T23:49:00.865051+0000","last_fullsized":"2024-04-24T23:49:00.865051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:31.412349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526184+0000","last_change":"2024-04-24T23:48:59.532080+0000","last_active":"2024-04-24T23:49:00.526184+0000","last_peered":"2024-04-24T23:49:00.526184+0000","last_clean":"2024-04-24T23:49:00.526184+0000","last_became_active":"2024-04-24T23:48:59.531298+0000","last_became_peered":"2024-04-24T23:48:59.531298+0000","last_unstale":"2024-04-24T23:49:00.526184+0000","last_undegraded":"2024-04-24T23:49:00.526184+0000","last_fullsized":"2024-04-24T23:49:00.526184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:39.512436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864909+0000","last_change":"2024-04-24T23:48:55.513313+0000","last_active":"2024-04-24T23:49:00.864909+0000","last_peered":"2024-04-24T23:49:00.864909+0000","last_clean":"2024-04-24T23:49:00.864909+0000","last_became_active":"2024-04-24T23:48:55.513021+0000","last_became_peered":"2024-04-24T23:48:55.513021+0000","last_unstale":"2024-04-24T23:49:00.864909+0000","last_undegraded":"2024-04-24T23:49:00.864909+0000","last_fullsized":"2024-04-24T23:49:00.864909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:04.801548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864612+0000","last_change":"2024-04-24T23:48:54.498900+0000","last_active":"2024-04-24T23:49:00.864612+0000","last_peered":"2024-04-24T23:49:00.864612+0000","last_clean":"2024-04-24T23:49:00.864612+0000","last_became_active":"2024-04-24T23:48:54.498604+0000","last_became_peered":"2024-04-24T23:48:54.498604+0000","last_unstale":"2024-04-24T23:49:00.864612+0000","last_undegraded":"2024-04-24T23:49:00.864612+0000","last_fullsized":"2024-04-24T23:49:00.864612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:15:10.269984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526077+0000","last_change":"2024-04-24T23:48:59.524031+0000","last_active":"2024-04-24T23:49:00.526077+0000","last_peered":"2024-04-24T23:49:00.526077+0000","last_clean":"2024-04-24T23:49:00.526077+0000","last_became_active":"2024-04-24T23:48:59.523585+0000","last_became_peered":"2024-04-24T23:48:59.523585+0000","last_unstale":"2024-04-24T23:49:00.526077+0000","last_undegraded":"2024-04-24T23:49:00.526077+0000","last_fullsized":"2024-04-24T23:49:00.526077+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:33.409892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863757+0000","last_change":"2024-04-24T23:48:59.539246+0000","last_active":"2024-04-24T23:49:00.863757+0000","last_peered":"2024-04-24T23:49:00.863757+0000","last_clean":"2024-04-24T23:49:00.863757+0000","last_became_active":"2024-04-24T23:48:59.538845+0000","last_became_peered":"2024-04-24T23:48:59.538845+0000","last_unstale":"2024-04-24T23:49:00.863757+0000","last_undegraded":"2024-04-24T23:49:00.863757+0000","last_fullsized":"2024-04-24T23:49:00.863757+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:37.247560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508807+0000","last_change":"2024-04-24T23:48:55.493299+0000","last_active":"2024-04-24T23:49:00.508807+0000","last_peered":"2024-04-24T23:49:00.508807+0000","last_clean":"2024-04-24T23:49:00.508807+0000","last_became_active":"2024-04-24T23:48:55.492944+0000","last_became_peered":"2024-04-24T23:48:55.492944+0000","last_unstale":"2024-04-24T23:49:00.508807+0000","last_undegraded":"2024-04-24T23:49:00.508807+0000","last_fullsized":"2024-04-24T23:49:00.508807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:53:38.263114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863928+0000","last_change":"2024-04-24T23:48:54.494727+0000","last_active":"2024-04-24T23:49:00.863928+0000","last_peered":"2024-04-24T23:49:00.863928+0000","last_clean":"2024-04-24T23:49:00.863928+0000","last_became_active":"2024-04-24T23:48:54.494441+0000","last_became_peered":"2024-04-24T23:48:54.494441+0000","last_unstale":"2024-04-24T23:49:00.863928+0000","last_undegraded":"2024-04-24T23:49:00.863928+0000","last_fullsized":"2024-04-24T23:49:00.863928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864551+0000","last_change":"2024-04-24T23:48:59.530686+0000","last_active":"2024-04-24T23:49:00.864551+0000","last_peered":"2024-04-24T23:49:00.864551+0000","last_clean":"2024-04-24T23:49:00.864551+0000","last_became_active":"2024-04-24T23:48:59.529705+0000","last_became_peered":"2024-04-24T23:48:59.529705+0000","last_unstale":"2024-04-24T23:49:00.864551+0000","last_undegraded":"2024-04-24T23:49:00.864551+0000","last_fullsized":"2024-04-24T23:49:00.864551+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:10.292844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526143+0000","last_change":"2024-04-24T23:48:55.498040+0000","last_active":"2024-04-24T23:49:00.526143+0000","last_peered":"2024-04-24T23:49:00.526143+0000","last_clean":"2024-04-24T23:49:00.526143+0000","last_became_active":"2024-04-24T23:48:55.497298+0000","last_became_peered":"2024-04-24T23:48:55.497298+0000","last_unstale":"2024-04-24T23:49:00.526143+0000","last_undegraded":"2024-04-24T23:49:00.526143+0000","last_fullsized":"2024-04-24T23:49:00.526143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526688+0000","last_change":"2024-04-24T23:48:54.495330+0000","last_active":"2024-04-24T23:49:00.526688+0000","last_peered":"2024-04-24T23:49:00.526688+0000","last_clean":"2024-04-24T23:49:00.526688+0000","last_became_active":"2024-04-24T23:48:54.495079+0000","last_became_peered":"2024-04-24T23:48:54.495079+0000","last_unstale":"2024-04-24T23:49:00.526688+0000","last_undegraded":"2024-04-24T23:49:00.526688+0000","last_fullsized":"2024-04-24T23:49:00.526688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:16:25.658826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864192+0000","last_change":"2024-04-24T23:48:54.492734+0000","last_active":"2024-04-24T23:49:00.864192+0000","last_peered":"2024-04-24T23:49:00.864192+0000","last_clean":"2024-04-24T23:49:00.864192+0000","last_became_active":"2024-04-24T23:48:54.492467+0000","last_became_peered":"2024-04-24T23:48:54.492467+0000","last_unstale":"2024-04-24T23:49:00.864192+0000","last_undegraded":"2024-04-24T23:49:00.864192+0000","last_fullsized":"2024-04-24T23:49:00.864192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864956+0000","last_change":"2024-04-24T23:48:55.507847+0000","last_active":"2024-04-24T23:49:00.864956+0000","last_peered":"2024-04-24T23:49:00.864956+0000","last_clean":"2024-04-24T23:49:00.864956+0000","last_became_active":"2024-04-24T23:48:55.507382+0000","last_became_peered":"2024-04-24T23:48:55.507382+0000","last_unstale":"2024-04-24T23:49:00.864956+0000","last_undegraded":"2024-04-24T23:49:00.864956+0000","last_fullsized":"2024-04-24T23:49:00.864956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863934+0000","last_change":"2024-04-24T23:48:59.521454+0000","last_active":"2024-04-24T23:49:00.863934+0000","last_peered":"2024-04-24T23:49:00.863934+0000","last_clean":"2024-04-24T23:49:00.863934+0000","last_became_active":"2024-04-24T23:48:59.521041+0000","last_became_peered":"2024-04-24T23:48:59.521041+0000","last_unstale":"2024-04-24T23:49:00.863934+0000","last_undegraded":"2024-04-24T23:49:00.863934+0000","last_fullsized":"2024-04-24T23:49:00.863934+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:42.836048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526295+0000","last_change":"2024-04-24T23:48:54.489072+0000","last_active":"2024-04-24T23:49:00.526295+0000","last_peered":"2024-04-24T23:49:00.526295+0000","last_clean":"2024-04-24T23:49:00.526295+0000","last_became_active":"2024-04-24T23:48:54.488735+0000","last_became_peered":"2024-04-24T23:48:54.488735+0000","last_unstale":"2024-04-24T23:49:00.526295+0000","last_undegraded":"2024-04-24T23:49:00.526295+0000","last_fullsized":"2024-04-24T23:49:00.526295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:26.511856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510340+0000","last_change":"2024-04-24T23:48:55.501095+0000","last_active":"2024-04-24T23:49:00.510340+0000","last_peered":"2024-04-24T23:49:00.510340+0000","last_clean":"2024-04-24T23:49:00.510340+0000","last_became_active":"2024-04-24T23:48:55.500804+0000","last_became_peered":"2024-04-24T23:48:55.500804+0000","last_unstale":"2024-04-24T23:49:00.510340+0000","last_undegraded":"2024-04-24T23:49:00.510340+0000","last_fullsized":"2024-04-24T23:49:00.510340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:15:11.274374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526847+0000","last_change":"2024-04-24T23:48:59.532387+0000","last_active":"2024-04-24T23:49:00.526847+0000","last_peered":"2024-04-24T23:49:00.526847+0000","last_clean":"2024-04-24T23:49:00.526847+0000","last_became_active":"2024-04-24T23:48:59.531805+0000","last_became_peered":"2024-04-24T23:48:59.531805+0000","last_unstale":"2024-04-24T23:49:00.526847+0000","last_undegraded":"2024-04-24T23:49:00.526847+0000","last_fullsized":"2024-04-24T23:49:00.526847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:10.597258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863811+0000","last_change":"2024-04-24T23:48:55.502131+0000","last_active":"2024-04-24T23:49:00.863811+0000","last_peered":"2024-04-24T23:49:00.863811+0000","last_clean":"2024-04-24T23:49:00.863811+0000","last_became_active":"2024-04-24T23:48:55.501871+0000","last_became_peered":"2024-04-24T23:48:55.501871+0000","last_unstale":"2024-04-24T23:49:00.863811+0000","last_undegraded":"2024-04-24T23:49:00.863811+0000","last_fullsized":"2024-04-24T23:49:00.863811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863465+0000","last_change":"2024-04-24T23:48:55.503813+0000","last_active":"2024-04-24T23:49:00.863465+0000","last_peered":"2024-04-24T23:49:00.863465+0000","last_clean":"2024-04-24T23:49:00.863465+0000","last_became_active":"2024-04-24T23:48:55.503517+0000","last_became_peered":"2024-04-24T23:48:55.503517+0000","last_unstale":"2024-04-24T23:49:00.863465+0000","last_undegraded":"2024-04-24T23:49:00.863465+0000","last_fullsized":"2024-04-24T23:49:00.863465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:25.555824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864441+0000","last_change":"2024-04-24T23:48:55.507595+0000","last_active":"2024-04-24T23:49:00.864441+0000","last_peered":"2024-04-24T23:49:00.864441+0000","last_clean":"2024-04-24T23:49:00.864441+0000","last_became_active":"2024-04-24T23:48:55.507307+0000","last_became_peered":"2024-04-24T23:48:55.507307+0000","last_unstale":"2024-04-24T23:49:00.864441+0000","last_undegraded":"2024-04-24T23:49:00.864441+0000","last_fullsized":"2024-04-24T23:49:00.864441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:10.290840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864867+0000","last_change":"2024-04-24T23:48:55.498028+0000","last_active":"2024-04-24T23:49:00.864867+0000","last_peered":"2024-04-24T23:49:00.864867+0000","last_clean":"2024-04-24T23:49:00.864867+0000","last_became_active":"2024-04-24T23:48:55.496715+0000","last_became_peered":"2024-04-24T23:48:55.496715+0000","last_unstale":"2024-04-24T23:49:00.864867+0000","last_undegraded":"2024-04-24T23:49:00.864867+0000","last_fullsized":"2024-04-24T23:49:00.864867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526777+0000","last_change":"2024-04-24T23:48:55.505719+0000","last_active":"2024-04-24T23:49:00.526777+0000","last_peered":"2024-04-24T23:49:00.526777+0000","last_clean":"2024-04-24T23:49:00.526777+0000","last_became_active":"2024-04-24T23:48:55.505418+0000","last_became_peered":"2024-04-24T23:48:55.505418+0000","last_unstale":"2024-04-24T23:49:00.526777+0000","last_undegraded":"2024-04-24T23:49:00.526777+0000","last_fullsized":"2024-04-24T23:49:00.526777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509327+0000","last_change":"2024-04-24T23:48:55.492403+0000","last_active":"2024-04-24T23:49:00.509327+0000","last_peered":"2024-04-24T23:49:00.509327+0000","last_clean":"2024-04-24T23:49:00.509327+0000","last_became_active":"2024-04-24T23:48:55.492112+0000","last_became_peered":"2024-04-24T23:48:55.492112+0000","last_unstale":"2024-04-24T23:49:00.509327+0000","last_undegraded":"2024-04-24T23:49:00.509327+0000","last_fullsized":"2024-04-24T23:49:00.509327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864731+0000","last_change":"2024-04-24T23:48:55.488480+0000","last_active":"2024-04-24T23:49:00.864731+0000","last_peered":"2024-04-24T23:49:00.864731+0000","last_clean":"2024-04-24T23:49:00.864731+0000","last_became_active":"2024-04-24T23:48:55.488134+0000","last_became_peered":"2024-04-24T23:48:55.488134+0000","last_unstale":"2024-04-24T23:49:00.864731+0000","last_undegraded":"2024-04-24T23:49:00.864731+0000","last_fullsized":"2024-04-24T23:49:00.864731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:55.758743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526760+0000","last_change":"2024-04-24T23:48:55.507589+0000","last_active":"2024-04-24T23:49:00.526760+0000","last_peered":"2024-04-24T23:49:00.526760+0000","last_clean":"2024-04-24T23:49:00.526760+0000","last_became_active":"2024-04-24T23:48:55.507281+0000","last_became_peered":"2024-04-24T23:48:55.507281+0000","last_unstale":"2024-04-24T23:49:00.526760+0000","last_undegraded":"2024-04-24T23:49:00.526760+0000","last_fullsized":"2024-04-24T23:49:00.526760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:51.930595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863920+0000","last_change":"2024-04-24T23:48:55.498917+0000","last_active":"2024-04-24T23:49:00.863920+0000","last_peered":"2024-04-24T23:49:00.863920+0000","last_clean":"2024-04-24T23:49:00.863920+0000","last_became_active":"2024-04-24T23:48:55.498627+0000","last_became_peered":"2024-04-24T23:48:55.498627+0000","last_unstale":"2024-04-24T23:49:00.863920+0000","last_undegraded":"2024-04-24T23:49:00.863920+0000","last_fullsized":"2024-04-24T23:49:00.863920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:55.758743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526124+0000","last_change":"2024-04-24T23:48:55.507449+0000","last_active":"2024-04-24T23:49:00.526124+0000","last_peered":"2024-04-24T23:49:00.526124+0000","last_clean":"2024-04-24T23:49:00.526124+0000","last_became_active":"2024-04-24T23:48:55.507151+0000","last_became_peered":"2024-04-24T23:48:55.507151+0000","last_unstale":"2024-04-24T23:49:00.526124+0000","last_undegraded":"2024-04-24T23:49:00.526124+0000","last_fullsized":"2024-04-24T23:49:00.526124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:46.889872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864334+0000","last_change":"2024-04-24T23:48:55.511263+0000","last_active":"2024-04-24T23:49:00.864334+0000","last_peered":"2024-04-24T23:49:00.864334+0000","last_clean":"2024-04-24T23:49:00.864334+0000","last_became_active":"2024-04-24T23:48:55.510422+0000","last_became_peered":"2024-04-24T23:48:55.510422+0000","last_unstale":"2024-04-24T23:49:00.864334+0000","last_undegraded":"2024-04-24T23:49:00.864334+0000","last_fullsized":"2024-04-24T23:49:00.864334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:47.627183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864769+0000","last_change":"2024-04-24T23:48:55.508355+0000","last_active":"2024-04-24T23:49:00.864769+0000","last_peered":"2024-04-24T23:49:00.864769+0000","last_clean":"2024-04-24T23:49:00.864769+0000","last_became_active":"2024-04-24T23:48:55.506836+0000","last_became_peered":"2024-04-24T23:48:55.506836+0000","last_unstale":"2024-04-24T23:49:00.864769+0000","last_undegraded":"2024-04-24T23:49:00.864769+0000","last_fullsized":"2024-04-24T23:49:00.864769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:53:38.263114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864802+0000","last_change":"2024-04-24T23:48:55.507774+0000","last_active":"2024-04-24T23:49:00.864802+0000","last_peered":"2024-04-24T23:49:00.864802+0000","last_clean":"2024-04-24T23:49:00.864802+0000","last_became_active":"2024-04-24T23:48:55.506816+0000","last_became_peered":"2024-04-24T23:48:55.506816+0000","last_unstale":"2024-04-24T23:49:00.864802+0000","last_undegraded":"2024-04-24T23:49:00.864802+0000","last_fullsized":"2024-04-24T23:49:00.864802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510851+0000","last_change":"2024-04-24T23:48:55.492334+0000","last_active":"2024-04-24T23:49:00.510851+0000","last_peered":"2024-04-24T23:49:00.510851+0000","last_clean":"2024-04-24T23:49:00.510851+0000","last_became_active":"2024-04-24T23:48:55.492065+0000","last_became_peered":"2024-04-24T23:48:55.492065+0000","last_unstale":"2024-04-24T23:49:00.510851+0000","last_undegraded":"2024-04-24T23:49:00.510851+0000","last_fullsized":"2024-04-24T23:49:00.510851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:42.483054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863835+0000","last_change":"2024-04-24T23:48:55.495084+0000","last_active":"2024-04-24T23:49:00.863835+0000","last_peered":"2024-04-24T23:49:00.863835+0000","last_clean":"2024-04-24T23:49:00.863835+0000","last_became_active":"2024-04-24T23:48:55.494776+0000","last_became_peered":"2024-04-24T23:48:55.494776+0000","last_unstale":"2024-04-24T23:49:00.863835+0000","last_undegraded":"2024-04-24T23:49:00.863835+0000","last_fullsized":"2024-04-24T23:49:00.863835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:32.416739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865234+0000","last_change":"2024-04-24T23:48:55.511141+0000","last_active":"2024-04-24T23:49:00.865234+0000","last_peered":"2024-04-24T23:49:00.865234+0000","last_clean":"2024-04-24T23:49:00.865234+0000","last_became_active":"2024-04-24T23:48:55.510836+0000","last_became_peered":"2024-04-24T23:48:55.510836+0000","last_unstale":"2024-04-24T23:49:00.865234+0000","last_undegraded":"2024-04-24T23:49:00.865234+0000","last_fullsized":"2024-04-24T23:49:00.865234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:18.684092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863787+0000","last_change":"2024-04-24T23:48:55.508632+0000","last_active":"2024-04-24T23:49:00.863787+0000","last_peered":"2024-04-24T23:49:00.863787+0000","last_clean":"2024-04-24T23:49:00.863787+0000","last_became_active":"2024-04-24T23:48:55.507448+0000","last_became_peered":"2024-04-24T23:48:55.507448+0000","last_unstale":"2024-04-24T23:49:00.863787+0000","last_undegraded":"2024-04-24T23:49:00.863787+0000","last_fullsized":"2024-04-24T23:49:00.863787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:46.889872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508815+0000","last_change":"2024-04-24T23:48:55.508181+0000","last_active":"2024-04-24T23:49:00.508815+0000","last_peered":"2024-04-24T23:49:00.508815+0000","last_clean":"2024-04-24T23:49:00.508815+0000","last_became_active":"2024-04-24T23:48:55.507877+0000","last_became_peered":"2024-04-24T23:48:55.507877+0000","last_unstale":"2024-04-24T23:49:00.508815+0000","last_undegraded":"2024-04-24T23:49:00.508815+0000","last_fullsized":"2024-04-24T23:49:00.508815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864457+0000","last_change":"2024-04-24T23:48:55.497468+0000","last_active":"2024-04-24T23:49:00.864457+0000","last_peered":"2024-04-24T23:49:00.864457+0000","last_clean":"2024-04-24T23:49:00.864457+0000","last_became_active":"2024-04-24T23:48:55.496754+0000","last_became_peered":"2024-04-24T23:48:55.496754+0000","last_unstale":"2024-04-24T23:49:00.864457+0000","last_undegraded":"2024-04-24T23:49:00.864457+0000","last_fullsized":"2024-04-24T23:49:00.864457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:58.209422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526144+0000","last_change":"2024-04-24T23:48:55.492306+0000","last_active":"2024-04-24T23:49:00.526144+0000","last_peered":"2024-04-24T23:49:00.526144+0000","last_clean":"2024-04-24T23:49:00.526144+0000","last_became_active":"2024-04-24T23:48:55.492008+0000","last_became_peered":"2024-04-24T23:48:55.492008+0000","last_unstale":"2024-04-24T23:49:00.526144+0000","last_undegraded":"2024-04-24T23:49:00.526144+0000","last_fullsized":"2024-04-24T23:49:00.526144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:55.758743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510496+0000","last_change":"2024-04-24T23:48:55.501767+0000","last_active":"2024-04-24T23:49:00.510496+0000","last_peered":"2024-04-24T23:49:00.510496+0000","last_clean":"2024-04-24T23:49:00.510496+0000","last_became_active":"2024-04-24T23:48:55.500776+0000","last_became_peered":"2024-04-24T23:48:55.500776+0000","last_unstale":"2024-04-24T23:49:00.510496+0000","last_undegraded":"2024-04-24T23:49:00.510496+0000","last_fullsized":"2024-04-24T23:49:00.510496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:04.085761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864571+0000","last_change":"2024-04-24T23:48:55.511220+0000","last_active":"2024-04-24T23:49:00.864571+0000","last_peered":"2024-04-24T23:49:00.864571+0000","last_clean":"2024-04-24T23:49:00.864571+0000","last_became_active":"2024-04-24T23:48:55.510937+0000","last_became_peered":"2024-04-24T23:48:55.510937+0000","last_unstale":"2024-04-24T23:49:00.864571+0000","last_undegraded":"2024-04-24T23:49:00.864571+0000","last_fullsized":"2024-04-24T23:49:00.864571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864622+0000","last_change":"2024-04-24T23:48:55.503843+0000","last_active":"2024-04-24T23:49:00.864622+0000","last_peered":"2024-04-24T23:49:00.864622+0000","last_clean":"2024-04-24T23:49:00.864622+0000","last_became_active":"2024-04-24T23:48:55.503569+0000","last_became_peered":"2024-04-24T23:48:55.503569+0000","last_unstale":"2024-04-24T23:49:00.864622+0000","last_undegraded":"2024-04-24T23:49:00.864622+0000","last_fullsized":"2024-04-24T23:49:00.864622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:46.889872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511357+0000","last_change":"2024-04-24T23:48:55.509090+0000","last_active":"2024-04-24T23:49:00.511357+0000","last_peered":"2024-04-24T23:49:00.511357+0000","last_clean":"2024-04-24T23:49:00.511357+0000","last_became_active":"2024-04-24T23:48:55.508730+0000","last_became_peered":"2024-04-24T23:48:55.508730+0000","last_unstale":"2024-04-24T23:49:00.511357+0000","last_undegraded":"2024-04-24T23:49:00.511357+0000","last_fullsized":"2024-04-24T23:49:00.511357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:18.684092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.525876+0000","last_change":"2024-04-24T23:48:55.498351+0000","last_active":"2024-04-24T23:49:00.525876+0000","last_peered":"2024-04-24T23:49:00.525876+0000","last_clean":"2024-04-24T23:49:00.525876+0000","last_became_active":"2024-04-24T23:48:55.497694+0000","last_became_peered":"2024-04-24T23:48:55.497694+0000","last_unstale":"2024-04-24T23:49:00.525876+0000","last_undegraded":"2024-04-24T23:49:00.525876+0000","last_fullsized":"2024-04-24T23:49:00.525876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509595+0000","last_change":"2024-04-24T23:48:55.501677+0000","last_active":"2024-04-24T23:49:00.509595+0000","last_peered":"2024-04-24T23:49:00.509595+0000","last_clean":"2024-04-24T23:49:00.509595+0000","last_became_active":"2024-04-24T23:48:55.501359+0000","last_became_peered":"2024-04-24T23:48:55.501359+0000","last_unstale":"2024-04-24T23:49:00.509595+0000","last_undegraded":"2024-04-24T23:49:00.509595+0000","last_fullsized":"2024-04-24T23:49:00.509595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:47.627183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864613+0000","last_change":"2024-04-24T23:48:55.511089+0000","last_active":"2024-04-24T23:49:00.864613+0000","last_peered":"2024-04-24T23:49:00.864613+0000","last_clean":"2024-04-24T23:49:00.864613+0000","last_became_active":"2024-04-24T23:48:55.510818+0000","last_became_peered":"2024-04-24T23:48:55.510818+0000","last_unstale":"2024-04-24T23:49:00.864613+0000","last_undegraded":"2024-04-24T23:49:00.864613+0000","last_fullsized":"2024-04-24T23:49:00.864613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:03:28.472120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509524+0000","last_change":"2024-04-24T23:48:55.492973+0000","last_active":"2024-04-24T23:49:00.509524+0000","last_peered":"2024-04-24T23:49:00.509524+0000","last_clean":"2024-04-24T23:49:00.509524+0000","last_became_active":"2024-04-24T23:48:55.492713+0000","last_became_peered":"2024-04-24T23:48:55.492713+0000","last_unstale":"2024-04-24T23:49:00.509524+0000","last_undegraded":"2024-04-24T23:49:00.509524+0000","last_fullsized":"2024-04-24T23:49:00.509524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:10.290840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864561+0000","last_change":"2024-04-24T23:48:55.508975+0000","last_active":"2024-04-24T23:49:00.864561+0000","last_peered":"2024-04-24T23:49:00.864561+0000","last_clean":"2024-04-24T23:49:00.864561+0000","last_became_active":"2024-04-24T23:48:55.507970+0000","last_became_peered":"2024-04-24T23:48:55.507970+0000","last_unstale":"2024-04-24T23:49:00.864561+0000","last_undegraded":"2024-04-24T23:49:00.864561+0000","last_fullsized":"2024-04-24T23:49:00.864561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:10.235664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864614+0000","last_change":"2024-04-24T23:48:55.500274+0000","last_active":"2024-04-24T23:49:00.864614+0000","last_peered":"2024-04-24T23:49:00.864614+0000","last_clean":"2024-04-24T23:49:00.864614+0000","last_became_active":"2024-04-24T23:48:55.500004+0000","last_became_peered":"2024-04-24T23:48:55.500004+0000","last_unstale":"2024-04-24T23:49:00.864614+0000","last_undegraded":"2024-04-24T23:49:00.864614+0000","last_fullsized":"2024-04-24T23:49:00.864614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:02:47.989664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509710+0000","last_change":"2024-04-24T23:48:55.499555+0000","last_active":"2024-04-24T23:49:00.509710+0000","last_peered":"2024-04-24T23:49:00.509710+0000","last_clean":"2024-04-24T23:49:00.509710+0000","last_became_active":"2024-04-24T23:48:55.499163+0000","last_became_peered":"2024-04-24T23:48:55.499163+0000","last_unstale":"2024-04-24T23:49:00.509710+0000","last_undegraded":"2024-04-24T23:49:00.509710+0000","last_fullsized":"2024-04-24T23:49:00.509710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511249+0000","last_change":"2024-04-24T23:48:55.493200+0000","last_active":"2024-04-24T23:49:00.511249+0000","last_peered":"2024-04-24T23:49:00.511249+0000","last_clean":"2024-04-24T23:49:00.511249+0000","last_became_active":"2024-04-24T23:48:55.491655+0000","last_became_peered":"2024-04-24T23:48:55.491655+0000","last_unstale":"2024-04-24T23:49:00.511249+0000","last_undegraded":"2024-04-24T23:49:00.511249+0000","last_fullsized":"2024-04-24T23:49:00.511249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:04.801548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:49:25.226 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:49:25.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.363+0000 7f45e0a8c640 1 Processor -- start 2024-04-24T23:49:25.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.363+0000 7f45e0a8c640 1 -- start start 2024-04-24T23:49:25.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.363+0000 7f45e0a8c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f45dc15e020 0x7f45dc0cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.363+0000 7f45e0a8c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f45dc0d1fa0 0x7f45dc0d23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.363+0000 7f45e0a8c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45dc0b3390 con 0x7f45dc0b2bb0 2024-04-24T23:49:25.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.363+0000 7f45e0a8c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f45dc06bfc0 con 0x7f45dc15e020 2024-04-24T23:49:25.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.363+0000 7f45e0a8c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f45dc06c2a0 con 0x7f45dc0d1fa0 2024-04-24T23:49:25.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.364+0000 7f45da59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f45dc15e020 0x7f45dc0cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.364+0000 7f45d3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f45dc0b2bb0 0x7f45dc0cf7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50740/0 (socket says 172.21.15.16:50740) 2024-04-24T23:49:25.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.364+0000 7f45d3fff640 1 -- 172.21.15.16:0/2480399125 learned_addr learned my addr 172.21.15.16:0/2480399125 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:25.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.364+0000 7f45da59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f45dc15e020 0x7f45dc0cf270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48728/0 (socket says 172.21.15.16:48728) 2024-04-24T23:49:25.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.364+0000 7f45dad9d640 1 --2- 172.21.15.16:0/2480399125 >> v2:172.21.15.156:3300/0 conn(0x7f45dc0d1fa0 0x7f45dc0d23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.364+0000 7f45da59c640 1 -- 172.21.15.16:0/2480399125 >> v2:172.21.15.156:3300/0 conn(0x7f45dc0d1fa0 msgr2=0x7f45dc0d23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:25.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.364+0000 7f45da59c640 1 --2- 172.21.15.16:0/2480399125 >> v2:172.21.15.156:3300/0 conn(0x7f45dc0d1fa0 0x7f45dc0d23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.364+0000 7f45da59c640 1 -- 172.21.15.16:0/2480399125 >> v1:172.21.15.16:6789/0 conn(0x7f45dc0b2bb0 legacy=0x7f45dc0cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.364+0000 7f45da59c640 1 -- 172.21.15.16:0/2480399125 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45dc06c580 con 0x7f45dc15e020 2024-04-24T23:49:25.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.364+0000 7f45dad9d640 1 --2- 172.21.15.16:0/2480399125 >> v2:172.21.15.156:3300/0 conn(0x7f45dc0d1fa0 0x7f45dc0d23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:25.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.365+0000 7f45da59c640 1 --2- 172.21.15.16:0/2480399125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45dc15e020 0x7f45dc0cf270 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0c78f0 tx=0x7f45cc02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b465c6163d28ee server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:25.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.365+0000 7f45d9d9b640 1 -- 172.21.15.16:0/2480399125 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc037070 con 0x7f45dc15e020 2024-04-24T23:49:25.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.365+0000 7f45d9d9b640 1 -- 172.21.15.16:0/2480399125 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45cc032440 con 0x7f45dc15e020 2024-04-24T23:49:25.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.365+0000 7f45d9d9b640 1 -- 172.21.15.16:0/2480399125 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc02ae60 con 0x7f45dc15e020 2024-04-24T23:49:25.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.365+0000 7f45e0a8c640 1 -- 172.21.15.16:0/2480399125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45dc15e020 msgr2=0x7f45dc0cf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.365+0000 7f45e0a8c640 1 --2- 172.21.15.16:0/2480399125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45dc15e020 0x7f45dc0cf270 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f45dc0c78f0 tx=0x7f45cc02a110 comp rx=0 tx=0).stop 2024-04-24T23:49:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.365+0000 7f45e0a8c640 1 -- 172.21.15.16:0/2480399125 shutdown_connections 2024-04-24T23:49:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.365+0000 7f45e0a8c640 1 --2- 172.21.15.16:0/2480399125 >> v2:172.21.15.156:3300/0 conn(0x7f45dc0d1fa0 0x7f45dc0d23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.365+0000 7f45e0a8c640 1 --2- 172.21.15.16:0/2480399125 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45dc15e020 0x7f45dc0cf270 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.365+0000 7f45e0a8c640 1 -- 172.21.15.16:0/2480399125 >> 172.21.15.16:0/2480399125 conn(0x7f45dc0d3e10 msgr2=0x7f45dc0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.365+0000 7f45e0a8c640 1 -- 172.21.15.16:0/2480399125 shutdown_connections 2024-04-24T23:49:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.365+0000 7f45e0a8c640 1 -- 172.21.15.16:0/2480399125 wait complete. 2024-04-24T23:49:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.367+0000 7f45e0a8c640 1 Processor -- start 2024-04-24T23:49:25.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.367+0000 7f45e0a8c640 1 -- start start 2024-04-24T23:49:25.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.367+0000 7f45e0a8c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45dc0b2bb0 0x7f45dc0b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.367+0000 7f45e0a8c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45dc0d1fa0 0x7f45dc149920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.367+0000 7f45e0a8c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45dc15e020 0x7f45dc149e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.367+0000 7f45e0a8c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc15e400 con 0x7f45dc15e020 2024-04-24T23:49:25.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.367+0000 7f45e0a8c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc0d1a80 con 0x7f45dc0b2bb0 2024-04-24T23:49:25.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.367+0000 7f45e0a8c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f45dc0cfec0 con 0x7f45dc0d1fa0 2024-04-24T23:49:25.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45dad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45dc15e020 0x7f45dc149e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45dad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45dc15e020 0x7f45dc149e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48736/0 (socket says 172.21.15.16:48736) 2024-04-24T23:49:25.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45dad9d640 1 -- 172.21.15.16:0/3139299573 learned_addr learned my addr 172.21.15.16:0/3139299573 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:25.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45da59c640 1 --2- 172.21.15.16:0/3139299573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45dc0b2bb0 0x7f45dc0b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45d3fff640 1 --2- 172.21.15.16:0/3139299573 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45dc0d1fa0 0x7f45dc149920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45dad9d640 1 -- 172.21.15.16:0/3139299573 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45dc0d1fa0 msgr2=0x7f45dc149920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45dad9d640 1 --2- 172.21.15.16:0/3139299573 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45dc0d1fa0 0x7f45dc149920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45dad9d640 1 -- 172.21.15.16:0/3139299573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45dc0b2bb0 msgr2=0x7f45dc0b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45dad9d640 1 --2- 172.21.15.16:0/3139299573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45dc0b2bb0 0x7f45dc0b4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45dad9d640 1 -- 172.21.15.16:0/3139299573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45dc14b4d0 con 0x7f45dc15e020 2024-04-24T23:49:25.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45dad9d640 1 --2- 172.21.15.16:0/3139299573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45dc15e020 0x7f45dc149e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f45d4007f20 tx=0x7f45d400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:25.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45d2ffd640 1 -- 172.21.15.16:0/3139299573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d401a070 con 0x7f45dc15e020 2024-04-24T23:49:25.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45d2ffd640 1 -- 172.21.15.16:0/3139299573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45d4003b90 con 0x7f45dc15e020 2024-04-24T23:49:25.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45e0a8c640 1 -- 172.21.15.16:0/3139299573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45dc14b7b0 con 0x7f45dc15e020 2024-04-24T23:49:25.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45d2ffd640 1 -- 172.21.15.16:0/3139299573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d4004030 con 0x7f45dc15e020 2024-04-24T23:49:25.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.368+0000 7f45e0a8c640 1 -- 172.21.15.16:0/3139299573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45dc14bd70 con 0x7f45dc15e020 2024-04-24T23:49:25.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.369+0000 7f45e0a8c640 1 -- 172.21.15.16:0/3139299573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45dc06dfe0 con 0x7f45dc15e020 2024-04-24T23:49:25.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.370+0000 7f45d2ffd640 1 -- 172.21.15.16:0/3139299573 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f45d401d070 con 0x7f45dc15e020 2024-04-24T23:49:25.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.370+0000 7f45d2ffd640 1 --2- 172.21.15.16:0/3139299573 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f45bc06be30 0x7f45bc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.370+0000 7f45d2ffd640 1 -- 172.21.15.16:0/3139299573 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f45d408b4c0 con 0x7f45dc15e020 2024-04-24T23:49:25.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.370+0000 7f45da59c640 1 --2- 172.21.15.16:0/3139299573 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f45bc06be30 0x7f45bc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.371+0000 7f45da59c640 1 --2- 172.21.15.16:0/3139299573 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f45bc06be30 0x7f45bc06e2f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f45cc034000 tx=0x7f45cc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:25.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.374+0000 7f45d2ffd640 1 -- 172.21.15.16:0/3139299573 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f45d4010070 con 0x7f45dc15e020 2024-04-24T23:49:25.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.554+0000 7f45e0a8c640 1 -- 172.21.15.16:0/3139299573 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f45dc0cdf10 con 0x7f45bc06be30 2024-04-24T23:49:25.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.563+0000 7f45d2ffd640 1 -- 172.21.15.16:0/3139299573 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+332531 (secure 0 0 0) 0x7f45dc0cdf10 con 0x7f45bc06be30 2024-04-24T23:49:25.565 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:25.567 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-24T23:49:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.566+0000 7f45e0a8c640 1 -- 172.21.15.16:0/3139299573 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f45bc06be30 msgr2=0x7f45bc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.566+0000 7f45e0a8c640 1 --2- 172.21.15.16:0/3139299573 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f45bc06be30 0x7f45bc06e2f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f45cc034000 tx=0x7f45cc02e000 comp rx=0 tx=0).stop 2024-04-24T23:49:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.566+0000 7f45e0a8c640 1 -- 172.21.15.16:0/3139299573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45dc15e020 msgr2=0x7f45dc149e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.566+0000 7f45e0a8c640 1 --2- 172.21.15.16:0/3139299573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45dc15e020 0x7f45dc149e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f45d4007f20 tx=0x7f45d400d560 comp rx=0 tx=0).stop 2024-04-24T23:49:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.566+0000 7f45e0a8c640 1 -- 172.21.15.16:0/3139299573 shutdown_connections 2024-04-24T23:49:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.566+0000 7f45e0a8c640 1 --2- 172.21.15.16:0/3139299573 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f45bc06be30 0x7f45bc06e2f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.566+0000 7f45e0a8c640 1 --2- 172.21.15.16:0/3139299573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45dc15e020 0x7f45dc149e60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.566+0000 7f45e0a8c640 1 --2- 172.21.15.16:0/3139299573 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45dc0d1fa0 0x7f45dc149920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.566+0000 7f45e0a8c640 1 --2- 172.21.15.16:0/3139299573 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45dc0b2bb0 0x7f45dc0b4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.566+0000 7f45e0a8c640 1 -- 172.21.15.16:0/3139299573 >> 172.21.15.16:0/3139299573 conn(0x7f45dc0d3e10 msgr2=0x7f45dc0b35a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.566+0000 7f45e0a8c640 1 -- 172.21.15.16:0/3139299573 shutdown_connections 2024-04-24T23:49:25.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.566+0000 7f45e0a8c640 1 -- 172.21.15.16:0/3139299573 wait complete. 2024-04-24T23:49:25.577 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-04-24T23:49:25.124160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595008,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":248620,"kb_used_data":3304,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":754726100,"statfs":{"total":773094113280,"available":772839526400,"internally_reserved":0,"allocated":3383296,"data_stored":2029432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003010"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":25,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864269+0000","last_change":"2024-04-24T23:48:54.496506+0000","last_active":"2024-04-24T23:49:00.864269+0000","last_peered":"2024-04-24T23:49:00.864269+0000","last_clean":"2024-04-24T23:49:00.864269+0000","last_became_active":"2024-04-24T23:48:54.496184+0000","last_became_peered":"2024-04-24T23:48:54.496184+0000","last_unstale":"2024-04-24T23:49:00.864269+0000","last_undegraded":"2024-04-24T23:49:00.864269+0000","last_fullsized":"2024-04-24T23:49:00.864269+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:26.511856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526502+0000","last_change":"2024-04-24T23:48:55.493918+0000","last_active":"2024-04-24T23:49:00.526502+0000","last_peered":"2024-04-24T23:49:00.526502+0000","last_clean":"2024-04-24T23:49:00.526502+0000","last_became_active":"2024-04-24T23:48:55.493534+0000","last_became_peered":"2024-04-24T23:48:55.493534+0000","last_unstale":"2024-04-24T23:49:00.526502+0000","last_undegraded":"2024-04-24T23:49:00.526502+0000","last_fullsized":"2024-04-24T23:49:00.526502+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:56:24.335063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.525963+0000","last_change":"2024-04-24T23:48:59.535521+0000","last_active":"2024-04-24T23:49:00.525963+0000","last_peered":"2024-04-24T23:49:00.525963+0000","last_clean":"2024-04-24T23:49:00.525963+0000","last_became_active":"2024-04-24T23:48:59.535120+0000","last_became_peered":"2024-04-24T23:48:59.535120+0000","last_unstale":"2024-04-24T23:49:00.525963+0000","last_undegraded":"2024-04-24T23:49:00.525963+0000","last_fullsized":"2024-04-24T23:49:00.525963+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:02:34.201356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526615+0000","last_change":"2024-04-24T23:48:54.494612+0000","last_active":"2024-04-24T23:49:00.526615+0000","last_peered":"2024-04-24T23:49:00.526615+0000","last_clean":"2024-04-24T23:49:00.526615+0000","last_became_active":"2024-04-24T23:48:54.494317+0000","last_became_peered":"2024-04-24T23:48:54.494317+0000","last_unstale":"2024-04-24T23:49:00.526615+0000","last_undegraded":"2024-04-24T23:49:00.526615+0000","last_fullsized":"2024-04-24T23:49:00.526615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:07.349136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864094+0000","last_change":"2024-04-24T23:48:55.512152+0000","last_active":"2024-04-24T23:49:00.864094+0000","last_peered":"2024-04-24T23:49:00.864094+0000","last_clean":"2024-04-24T23:49:00.864094+0000","last_became_active":"2024-04-24T23:48:55.511396+0000","last_became_peered":"2024-04-24T23:48:55.511396+0000","last_unstale":"2024-04-24T23:49:00.864094+0000","last_undegraded":"2024-04-24T23:49:00.864094+0000","last_fullsized":"2024-04-24T23:49:00.864094+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510457+0000","last_change":"2024-04-24T23:48:59.530584+0000","last_active":"2024-04-24T23:49:00.510457+0000","last_peered":"2024-04-24T23:49:00.510457+0000","last_clean":"2024-04-24T23:49:00.510457+0000","last_became_active":"2024-04-24T23:48:59.530081+0000","last_became_peered":"2024-04-24T23:48:59.530081+0000","last_unstale":"2024-04-24T23:49:00.510457+0000","last_undegraded":"2024-04-24T23:49:00.510457+0000","last_fullsized":"2024-04-24T23:49:00.510457+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:03:32.497072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510851+0000","last_change":"2024-04-24T23:48:54.495374+0000","last_active":"2024-04-24T23:49:00.510851+0000","last_peered":"2024-04-24T23:49:00.510851+0000","last_clean":"2024-04-24T23:49:00.510851+0000","last_became_active":"2024-04-24T23:48:54.495040+0000","last_became_peered":"2024-04-24T23:48:54.495040+0000","last_unstale":"2024-04-24T23:49:00.510851+0000","last_undegraded":"2024-04-24T23:49:00.510851+0000","last_fullsized":"2024-04-24T23:49:00.510851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:31.412349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864681+0000","last_change":"2024-04-24T23:48:55.505099+0000","last_active":"2024-04-24T23:49:00.864681+0000","last_peered":"2024-04-24T23:49:00.864681+0000","last_clean":"2024-04-24T23:49:00.864681+0000","last_became_active":"2024-04-24T23:48:55.504778+0000","last_became_peered":"2024-04-24T23:48:55.504778+0000","last_unstale":"2024-04-24T23:49:00.864681+0000","last_undegraded":"2024-04-24T23:49:00.864681+0000","last_fullsized":"2024-04-24T23:49:00.864681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:17.488901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511291+0000","last_change":"2024-04-24T23:48:59.526537+0000","last_active":"2024-04-24T23:49:00.511291+0000","last_peered":"2024-04-24T23:49:00.511291+0000","last_clean":"2024-04-24T23:49:00.511291+0000","last_became_active":"2024-04-24T23:48:59.526038+0000","last_became_peered":"2024-04-24T23:48:59.526038+0000","last_unstale":"2024-04-24T23:49:00.511291+0000","last_undegraded":"2024-04-24T23:49:00.511291+0000","last_fullsized":"2024-04-24T23:49:00.511291+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510029+0000","last_change":"2024-04-24T23:48:54.498311+0000","last_active":"2024-04-24T23:49:00.510029+0000","last_peered":"2024-04-24T23:49:00.510029+0000","last_clean":"2024-04-24T23:49:00.510029+0000","last_became_active":"2024-04-24T23:48:54.497986+0000","last_became_peered":"2024-04-24T23:48:54.497986+0000","last_unstale":"2024-04-24T23:49:00.510029+0000","last_undegraded":"2024-04-24T23:49:00.510029+0000","last_fullsized":"2024-04-24T23:49:00.510029+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865047+0000","last_change":"2024-04-24T23:48:55.507029+0000","last_active":"2024-04-24T23:49:00.865047+0000","last_peered":"2024-04-24T23:49:00.865047+0000","last_clean":"2024-04-24T23:49:00.865047+0000","last_became_active":"2024-04-24T23:48:55.506772+0000","last_became_peered":"2024-04-24T23:48:55.506772+0000","last_unstale":"2024-04-24T23:49:00.865047+0000","last_undegraded":"2024-04-24T23:49:00.865047+0000","last_fullsized":"2024-04-24T23:49:00.865047+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:42.483054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509920+0000","last_change":"2024-04-24T23:48:59.530333+0000","last_active":"2024-04-24T23:49:00.509920+0000","last_peered":"2024-04-24T23:49:00.509920+0000","last_clean":"2024-04-24T23:49:00.509920+0000","last_became_active":"2024-04-24T23:48:59.529831+0000","last_became_peered":"2024-04-24T23:48:59.529831+0000","last_unstale":"2024-04-24T23:49:00.509920+0000","last_undegraded":"2024-04-24T23:49:00.509920+0000","last_fullsized":"2024-04-24T23:49:00.509920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:47.852102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864057+0000","last_change":"2024-04-24T23:48:55.504704+0000","last_active":"2024-04-24T23:49:00.864057+0000","last_peered":"2024-04-24T23:49:00.864057+0000","last_clean":"2024-04-24T23:49:00.864057+0000","last_became_active":"2024-04-24T23:48:55.504444+0000","last_became_peered":"2024-04-24T23:48:55.504444+0000","last_unstale":"2024-04-24T23:49:00.864057+0000","last_undegraded":"2024-04-24T23:49:00.864057+0000","last_fullsized":"2024-04-24T23:49:00.864057+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510280+0000","last_change":"2024-04-24T23:48:54.493270+0000","last_active":"2024-04-24T23:49:00.510280+0000","last_peered":"2024-04-24T23:49:00.510280+0000","last_clean":"2024-04-24T23:49:00.510280+0000","last_became_active":"2024-04-24T23:48:54.492952+0000","last_became_peered":"2024-04-24T23:48:54.492952+0000","last_unstale":"2024-04-24T23:49:00.510280+0000","last_undegraded":"2024-04-24T23:49:00.510280+0000","last_fullsized":"2024-04-24T23:49:00.510280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:36:57.277988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510165+0000","last_change":"2024-04-24T23:48:59.530237+0000","last_active":"2024-04-24T23:49:00.510165+0000","last_peered":"2024-04-24T23:49:00.510165+0000","last_clean":"2024-04-24T23:49:00.510165+0000","last_became_active":"2024-04-24T23:48:59.529732+0000","last_became_peered":"2024-04-24T23:48:59.529732+0000","last_unstale":"2024-04-24T23:49:00.510165+0000","last_undegraded":"2024-04-24T23:49:00.510165+0000","last_fullsized":"2024-04-24T23:49:00.510165+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:59.783695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526172+0000","last_change":"2024-04-24T23:48:55.500061+0000","last_active":"2024-04-24T23:49:00.526172+0000","last_peered":"2024-04-24T23:49:00.526172+0000","last_clean":"2024-04-24T23:49:00.526172+0000","last_became_active":"2024-04-24T23:48:55.499515+0000","last_became_peered":"2024-04-24T23:48:55.499515+0000","last_unstale":"2024-04-24T23:49:00.526172+0000","last_undegraded":"2024-04-24T23:49:00.526172+0000","last_fullsized":"2024-04-24T23:49:00.526172+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:10.235664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509506+0000","last_change":"2024-04-24T23:48:54.496643+0000","last_active":"2024-04-24T23:49:00.509506+0000","last_peered":"2024-04-24T23:49:00.509506+0000","last_clean":"2024-04-24T23:49:00.509506+0000","last_became_active":"2024-04-24T23:48:54.496367+0000","last_became_peered":"2024-04-24T23:48:54.496367+0000","last_unstale":"2024-04-24T23:49:00.509506+0000","last_undegraded":"2024-04-24T23:49:00.509506+0000","last_fullsized":"2024-04-24T23:49:00.509506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:17.375124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510348+0000","last_change":"2024-04-24T23:48:59.531327+0000","last_active":"2024-04-24T23:49:00.510348+0000","last_peered":"2024-04-24T23:49:00.510348+0000","last_clean":"2024-04-24T23:49:00.510348+0000","last_became_active":"2024-04-24T23:48:59.530959+0000","last_became_peered":"2024-04-24T23:48:59.530959+0000","last_unstale":"2024-04-24T23:49:00.510348+0000","last_undegraded":"2024-04-24T23:49:00.510348+0000","last_fullsized":"2024-04-24T23:49:00.510348+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:14.260616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526263+0000","last_change":"2024-04-24T23:48:55.494442+0000","last_active":"2024-04-24T23:49:00.526263+0000","last_peered":"2024-04-24T23:49:00.526263+0000","last_clean":"2024-04-24T23:49:00.526263+0000","last_became_active":"2024-04-24T23:48:55.494165+0000","last_became_peered":"2024-04-24T23:48:55.494165+0000","last_unstale":"2024-04-24T23:49:00.526263+0000","last_undegraded":"2024-04-24T23:49:00.526263+0000","last_fullsized":"2024-04-24T23:49:00.526263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508952+0000","last_change":"2024-04-24T23:48:54.488761+0000","last_active":"2024-04-24T23:49:00.508952+0000","last_peered":"2024-04-24T23:49:00.508952+0000","last_clean":"2024-04-24T23:49:00.508952+0000","last_became_active":"2024-04-24T23:48:54.488431+0000","last_became_peered":"2024-04-24T23:48:54.488431+0000","last_unstale":"2024-04-24T23:49:00.508952+0000","last_undegraded":"2024-04-24T23:49:00.508952+0000","last_fullsized":"2024-04-24T23:49:00.508952+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509720+0000","last_change":"2024-04-24T23:48:59.523973+0000","last_active":"2024-04-24T23:49:00.509720+0000","last_peered":"2024-04-24T23:49:00.509720+0000","last_clean":"2024-04-24T23:49:00.509720+0000","last_became_active":"2024-04-24T23:48:59.520679+0000","last_became_peered":"2024-04-24T23:48:59.520679+0000","last_unstale":"2024-04-24T23:49:00.509720+0000","last_undegraded":"2024-04-24T23:49:00.509720+0000","last_fullsized":"2024-04-24T23:49:00.509720+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511072+0000","last_change":"2024-04-24T23:48:55.505618+0000","last_active":"2024-04-24T23:49:00.511072+0000","last_peered":"2024-04-24T23:49:00.511072+0000","last_clean":"2024-04-24T23:49:00.511072+0000","last_became_active":"2024-04-24T23:48:55.504804+0000","last_became_peered":"2024-04-24T23:48:55.504804+0000","last_unstale":"2024-04-24T23:49:00.511072+0000","last_undegraded":"2024-04-24T23:49:00.511072+0000","last_fullsized":"2024-04-24T23:49:00.511072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:46.215368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509409+0000","last_change":"2024-04-24T23:48:54.496898+0000","last_active":"2024-04-24T23:49:00.509409+0000","last_peered":"2024-04-24T23:49:00.509409+0000","last_clean":"2024-04-24T23:49:00.509409+0000","last_became_active":"2024-04-24T23:48:54.496603+0000","last_became_peered":"2024-04-24T23:48:54.496603+0000","last_unstale":"2024-04-24T23:49:00.509409+0000","last_undegraded":"2024-04-24T23:49:00.509409+0000","last_fullsized":"2024-04-24T23:49:00.509409+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510695+0000","last_change":"2024-04-24T23:48:59.533868+0000","last_active":"2024-04-24T23:49:00.510695+0000","last_peered":"2024-04-24T23:49:00.510695+0000","last_clean":"2024-04-24T23:49:00.510695+0000","last_became_active":"2024-04-24T23:48:59.533487+0000","last_became_peered":"2024-04-24T23:48:59.533487+0000","last_unstale":"2024-04-24T23:49:00.510695+0000","last_undegraded":"2024-04-24T23:49:00.510695+0000","last_fullsized":"2024-04-24T23:49:00.510695+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:21.513853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864071+0000","last_change":"2024-04-24T23:48:55.497734+0000","last_active":"2024-04-24T23:49:00.864071+0000","last_peered":"2024-04-24T23:49:00.864071+0000","last_clean":"2024-04-24T23:49:00.864071+0000","last_became_active":"2024-04-24T23:48:55.496634+0000","last_became_peered":"2024-04-24T23:48:55.496634+0000","last_unstale":"2024-04-24T23:49:00.864071+0000","last_undegraded":"2024-04-24T23:49:00.864071+0000","last_fullsized":"2024-04-24T23:49:00.864071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:56:24.335063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864834+0000","last_change":"2024-04-24T23:48:54.499625+0000","last_active":"2024-04-24T23:49:00.864834+0000","last_peered":"2024-04-24T23:49:00.864834+0000","last_clean":"2024-04-24T23:49:00.864834+0000","last_became_active":"2024-04-24T23:48:54.499357+0000","last_became_peered":"2024-04-24T23:48:54.499357+0000","last_unstale":"2024-04-24T23:49:00.864834+0000","last_undegraded":"2024-04-24T23:49:00.864834+0000","last_fullsized":"2024-04-24T23:49:00.864834+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511165+0000","last_change":"2024-04-24T23:48:59.523817+0000","last_active":"2024-04-24T23:49:00.511165+0000","last_peered":"2024-04-24T23:49:00.511165+0000","last_clean":"2024-04-24T23:49:00.511165+0000","last_became_active":"2024-04-24T23:48:59.522505+0000","last_became_peered":"2024-04-24T23:48:59.522505+0000","last_unstale":"2024-04-24T23:49:00.511165+0000","last_undegraded":"2024-04-24T23:49:00.511165+0000","last_fullsized":"2024-04-24T23:49:00.511165+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:37.247560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526836+0000","last_change":"2024-04-24T23:48:55.490605+0000","last_active":"2024-04-24T23:49:00.526836+0000","last_peered":"2024-04-24T23:49:00.526836+0000","last_clean":"2024-04-24T23:49:00.526836+0000","last_became_active":"2024-04-24T23:48:55.490271+0000","last_became_peered":"2024-04-24T23:48:55.490271+0000","last_unstale":"2024-04-24T23:49:00.526836+0000","last_undegraded":"2024-04-24T23:49:00.526836+0000","last_fullsized":"2024-04-24T23:49:00.526836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:53:38.263114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509803+0000","last_change":"2024-04-24T23:48:54.493101+0000","last_active":"2024-04-24T23:49:00.509803+0000","last_peered":"2024-04-24T23:49:00.509803+0000","last_clean":"2024-04-24T23:49:00.509803+0000","last_became_active":"2024-04-24T23:48:54.492768+0000","last_became_peered":"2024-04-24T23:48:54.492768+0000","last_unstale":"2024-04-24T23:49:00.509803+0000","last_undegraded":"2024-04-24T23:49:00.509803+0000","last_fullsized":"2024-04-24T23:49:00.509803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:15:10.269984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.525949+0000","last_change":"2024-04-24T23:48:59.535686+0000","last_active":"2024-04-24T23:49:00.525949+0000","last_peered":"2024-04-24T23:49:00.525949+0000","last_clean":"2024-04-24T23:49:00.525949+0000","last_became_active":"2024-04-24T23:48:59.535320+0000","last_became_peered":"2024-04-24T23:48:59.535320+0000","last_unstale":"2024-04-24T23:49:00.525949+0000","last_undegraded":"2024-04-24T23:49:00.525949+0000","last_fullsized":"2024-04-24T23:49:00.525949+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:22.709044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509618+0000","last_change":"2024-04-24T23:48:55.493935+0000","last_active":"2024-04-24T23:49:00.509618+0000","last_peered":"2024-04-24T23:49:00.509618+0000","last_clean":"2024-04-24T23:49:00.509618+0000","last_became_active":"2024-04-24T23:48:55.493604+0000","last_became_peered":"2024-04-24T23:48:55.493604+0000","last_unstale":"2024-04-24T23:49:00.509618+0000","last_undegraded":"2024-04-24T23:49:00.509618+0000","last_fullsized":"2024-04-24T23:49:00.509618+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510971+0000","last_change":"2024-04-24T23:48:54.497633+0000","last_active":"2024-04-24T23:49:00.510971+0000","last_peered":"2024-04-24T23:49:00.510971+0000","last_clean":"2024-04-24T23:49:00.510971+0000","last_became_active":"2024-04-24T23:48:54.497329+0000","last_became_peered":"2024-04-24T23:48:54.497329+0000","last_unstale":"2024-04-24T23:49:00.510971+0000","last_undegraded":"2024-04-24T23:49:00.510971+0000","last_fullsized":"2024-04-24T23:49:00.510971+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:37.806706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864823+0000","last_change":"2024-04-24T23:48:59.538720+0000","last_active":"2024-04-24T23:49:00.864823+0000","last_peered":"2024-04-24T23:49:00.864823+0000","last_clean":"2024-04-24T23:49:00.864823+0000","last_became_active":"2024-04-24T23:48:59.538310+0000","last_became_peered":"2024-04-24T23:48:59.538310+0000","last_unstale":"2024-04-24T23:49:00.864823+0000","last_undegraded":"2024-04-24T23:49:00.864823+0000","last_fullsized":"2024-04-24T23:49:00.864823+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:10.292844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865110+0000","last_change":"2024-04-24T23:48:55.510066+0000","last_active":"2024-04-24T23:49:00.865110+0000","last_peered":"2024-04-24T23:49:00.865110+0000","last_clean":"2024-04-24T23:49:00.865110+0000","last_became_active":"2024-04-24T23:48:55.507977+0000","last_became_peered":"2024-04-24T23:48:55.507977+0000","last_unstale":"2024-04-24T23:49:00.865110+0000","last_undegraded":"2024-04-24T23:49:00.865110+0000","last_fullsized":"2024-04-24T23:49:00.865110+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:06.267893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509767+0000","last_change":"2024-04-24T23:48:54.493032+0000","last_active":"2024-04-24T23:49:00.509767+0000","last_peered":"2024-04-24T23:49:00.509767+0000","last_clean":"2024-04-24T23:49:00.509767+0000","last_became_active":"2024-04-24T23:48:54.492694+0000","last_became_peered":"2024-04-24T23:48:54.492694+0000","last_unstale":"2024-04-24T23:49:00.509767+0000","last_undegraded":"2024-04-24T23:49:00.509767+0000","last_fullsized":"2024-04-24T23:49:00.509767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509213+0000","last_change":"2024-04-24T23:48:59.527964+0000","last_active":"2024-04-24T23:49:00.509213+0000","last_peered":"2024-04-24T23:49:00.509213+0000","last_clean":"2024-04-24T23:49:00.509213+0000","last_became_active":"2024-04-24T23:48:59.527519+0000","last_became_peered":"2024-04-24T23:48:59.527519+0000","last_unstale":"2024-04-24T23:49:00.509213+0000","last_undegraded":"2024-04-24T23:49:00.509213+0000","last_fullsized":"2024-04-24T23:49:00.509213+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:37.247560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511053+0000","last_change":"2024-04-24T23:48:55.505703+0000","last_active":"2024-04-24T23:49:00.511053+0000","last_peered":"2024-04-24T23:49:00.511053+0000","last_clean":"2024-04-24T23:49:00.511053+0000","last_became_active":"2024-04-24T23:48:55.505257+0000","last_became_peered":"2024-04-24T23:48:55.505257+0000","last_unstale":"2024-04-24T23:49:00.511053+0000","last_undegraded":"2024-04-24T23:49:00.511053+0000","last_fullsized":"2024-04-24T23:49:00.511053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:41.372867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864734+0000","last_change":"2024-04-24T23:48:54.494961+0000","last_active":"2024-04-24T23:49:00.864734+0000","last_peered":"2024-04-24T23:49:00.864734+0000","last_clean":"2024-04-24T23:49:00.864734+0000","last_became_active":"2024-04-24T23:48:54.494672+0000","last_became_peered":"2024-04-24T23:48:54.494672+0000","last_unstale":"2024-04-24T23:49:00.864734+0000","last_undegraded":"2024-04-24T23:49:00.864734+0000","last_fullsized":"2024-04-24T23:49:00.864734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509770+0000","last_change":"2024-04-24T23:48:59.525981+0000","last_active":"2024-04-24T23:49:00.509770+0000","last_peered":"2024-04-24T23:49:00.509770+0000","last_clean":"2024-04-24T23:49:00.509770+0000","last_became_active":"2024-04-24T23:48:59.524788+0000","last_became_peered":"2024-04-24T23:48:59.524788+0000","last_unstale":"2024-04-24T23:49:00.509770+0000","last_undegraded":"2024-04-24T23:49:00.509770+0000","last_fullsized":"2024-04-24T23:49:00.509770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:08.826500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508934+0000","last_change":"2024-04-24T23:48:55.493966+0000","last_active":"2024-04-24T23:49:00.508934+0000","last_peered":"2024-04-24T23:49:00.508934+0000","last_clean":"2024-04-24T23:49:00.508934+0000","last_became_active":"2024-04-24T23:48:55.493622+0000","last_became_peered":"2024-04-24T23:48:55.493622+0000","last_unstale":"2024-04-24T23:49:00.508934+0000","last_undegraded":"2024-04-24T23:49:00.508934+0000","last_fullsized":"2024-04-24T23:49:00.508934+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863565+0000","last_change":"2024-04-24T23:48:54.497811+0000","last_active":"2024-04-24T23:49:00.863565+0000","last_peered":"2024-04-24T23:49:00.863565+0000","last_clean":"2024-04-24T23:49:00.863565+0000","last_became_active":"2024-04-24T23:48:54.497465+0000","last_became_peered":"2024-04-24T23:48:54.497465+0000","last_unstale":"2024-04-24T23:49:00.863565+0000","last_undegraded":"2024-04-24T23:49:00.863565+0000","last_fullsized":"2024-04-24T23:49:00.863565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:50:17.375124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509953+0000","last_change":"2024-04-24T23:48:59.536596+0000","last_active":"2024-04-24T23:49:00.509953+0000","last_peered":"2024-04-24T23:49:00.509953+0000","last_clean":"2024-04-24T23:49:00.509953+0000","last_became_active":"2024-04-24T23:48:59.536186+0000","last_became_peered":"2024-04-24T23:48:59.536186+0000","last_unstale":"2024-04-24T23:49:00.509953+0000","last_undegraded":"2024-04-24T23:49:00.509953+0000","last_fullsized":"2024-04-24T23:49:00.509953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:46.508006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863695+0000","last_change":"2024-04-24T23:48:55.507490+0000","last_active":"2024-04-24T23:49:00.863695+0000","last_peered":"2024-04-24T23:49:00.863695+0000","last_clean":"2024-04-24T23:49:00.863695+0000","last_became_active":"2024-04-24T23:48:55.506415+0000","last_became_peered":"2024-04-24T23:48:55.506415+0000","last_unstale":"2024-04-24T23:49:00.863695+0000","last_undegraded":"2024-04-24T23:49:00.863695+0000","last_fullsized":"2024-04-24T23:49:00.863695+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509313+0000","last_change":"2024-04-24T23:48:54.491388+0000","last_active":"2024-04-24T23:49:00.509313+0000","last_peered":"2024-04-24T23:49:00.509313+0000","last_clean":"2024-04-24T23:49:00.509313+0000","last_became_active":"2024-04-24T23:48:54.490991+0000","last_became_peered":"2024-04-24T23:48:54.490991+0000","last_unstale":"2024-04-24T23:49:00.509313+0000","last_undegraded":"2024-04-24T23:49:00.509313+0000","last_fullsized":"2024-04-24T23:49:00.509313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863944+0000","last_change":"2024-04-24T23:48:59.539922+0000","last_active":"2024-04-24T23:49:00.863944+0000","last_peered":"2024-04-24T23:49:00.863944+0000","last_clean":"2024-04-24T23:49:00.863944+0000","last_became_active":"2024-04-24T23:48:59.539449+0000","last_became_peered":"2024-04-24T23:48:59.539449+0000","last_unstale":"2024-04-24T23:49:00.863944+0000","last_undegraded":"2024-04-24T23:49:00.863944+0000","last_fullsized":"2024-04-24T23:49:00.863944+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:08.826500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526078+0000","last_change":"2024-04-24T23:48:55.509237+0000","last_active":"2024-04-24T23:49:00.526078+0000","last_peered":"2024-04-24T23:49:00.526078+0000","last_clean":"2024-04-24T23:49:00.526078+0000","last_became_active":"2024-04-24T23:48:55.508847+0000","last_became_peered":"2024-04-24T23:48:55.508847+0000","last_unstale":"2024-04-24T23:49:00.526078+0000","last_undegraded":"2024-04-24T23:49:00.526078+0000","last_fullsized":"2024-04-24T23:49:00.526078+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864955+0000","last_change":"2024-04-24T23:48:54.496310+0000","last_active":"2024-04-24T23:49:00.864955+0000","last_peered":"2024-04-24T23:49:00.864955+0000","last_clean":"2024-04-24T23:49:00.864955+0000","last_became_active":"2024-04-24T23:48:54.496016+0000","last_became_peered":"2024-04-24T23:48:54.496016+0000","last_unstale":"2024-04-24T23:49:00.864955+0000","last_undegraded":"2024-04-24T23:49:00.864955+0000","last_fullsized":"2024-04-24T23:49:00.864955+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509522+0000","last_change":"2024-04-24T23:48:45.082459+0000","last_active":"2024-04-24T23:49:00.509522+0000","last_peered":"2024-04-24T23:49:00.509522+0000","last_clean":"2024-04-24T23:49:00.509522+0000","last_became_active":"2024-04-24T23:48:45.081495+0000","last_became_peered":"2024-04-24T23:48:45.081495+0000","last_unstale":"2024-04-24T23:49:00.509522+0000","last_undegraded":"2024-04-24T23:49:00.509522+0000","last_fullsized":"2024-04-24T23:49:00.509522+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_clean_scrub_stamp":"2024-04-24T23:48:43.440993+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:53:20.832511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864431+0000","last_change":"2024-04-24T23:48:59.521532+0000","last_active":"2024-04-24T23:49:00.864431+0000","last_peered":"2024-04-24T23:49:00.864431+0000","last_clean":"2024-04-24T23:49:00.864431+0000","last_became_active":"2024-04-24T23:48:59.521144+0000","last_became_peered":"2024-04-24T23:48:59.521144+0000","last_unstale":"2024-04-24T23:49:00.864431+0000","last_undegraded":"2024-04-24T23:49:00.864431+0000","last_fullsized":"2024-04-24T23:49:00.864431+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:08.826500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510601+0000","last_change":"2024-04-24T23:48:55.509123+0000","last_active":"2024-04-24T23:49:00.510601+0000","last_peered":"2024-04-24T23:49:00.510601+0000","last_clean":"2024-04-24T23:49:00.510601+0000","last_became_active":"2024-04-24T23:48:55.508860+0000","last_became_peered":"2024-04-24T23:48:55.508860+0000","last_unstale":"2024-04-24T23:49:00.510601+0000","last_undegraded":"2024-04-24T23:49:00.510601+0000","last_fullsized":"2024-04-24T23:49:00.510601+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:03:28.472120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863884+0000","last_change":"2024-04-24T23:48:54.498741+0000","last_active":"2024-04-24T23:49:00.863884+0000","last_peered":"2024-04-24T23:49:00.863884+0000","last_clean":"2024-04-24T23:49:00.863884+0000","last_became_active":"2024-04-24T23:48:54.498241+0000","last_became_peered":"2024-04-24T23:48:54.498241+0000","last_unstale":"2024-04-24T23:49:00.863884+0000","last_undegraded":"2024-04-24T23:49:00.863884+0000","last_fullsized":"2024-04-24T23:49:00.863884+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:57:07.349136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864020+0000","last_change":"2024-04-24T23:48:59.528434+0000","last_active":"2024-04-24T23:49:00.864020+0000","last_peered":"2024-04-24T23:49:00.864020+0000","last_clean":"2024-04-24T23:49:00.864020+0000","last_became_active":"2024-04-24T23:48:59.528062+0000","last_became_peered":"2024-04-24T23:48:59.528062+0000","last_unstale":"2024-04-24T23:49:00.864020+0000","last_undegraded":"2024-04-24T23:49:00.864020+0000","last_fullsized":"2024-04-24T23:49:00.864020+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864801+0000","last_change":"2024-04-24T23:48:55.509799+0000","last_active":"2024-04-24T23:49:00.864801+0000","last_peered":"2024-04-24T23:49:00.864801+0000","last_clean":"2024-04-24T23:49:00.864801+0000","last_became_active":"2024-04-24T23:48:55.509286+0000","last_became_peered":"2024-04-24T23:48:55.509286+0000","last_unstale":"2024-04-24T23:49:00.864801+0000","last_undegraded":"2024-04-24T23:49:00.864801+0000","last_fullsized":"2024-04-24T23:49:00.864801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:33.222608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509751+0000","last_change":"2024-04-24T23:48:54.490614+0000","last_active":"2024-04-24T23:49:00.509751+0000","last_peered":"2024-04-24T23:49:00.509751+0000","last_clean":"2024-04-24T23:49:00.509751+0000","last_became_active":"2024-04-24T23:48:54.490029+0000","last_became_peered":"2024-04-24T23:48:54.490029+0000","last_unstale":"2024-04-24T23:49:00.509751+0000","last_undegraded":"2024-04-24T23:49:00.509751+0000","last_fullsized":"2024-04-24T23:49:00.509751+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864962+0000","last_change":"2024-04-24T23:48:59.529966+0000","last_active":"2024-04-24T23:49:00.864962+0000","last_peered":"2024-04-24T23:49:00.864962+0000","last_clean":"2024-04-24T23:49:00.864962+0000","last_became_active":"2024-04-24T23:48:59.529508+0000","last_became_peered":"2024-04-24T23:48:59.529508+0000","last_unstale":"2024-04-24T23:49:00.864962+0000","last_undegraded":"2024-04-24T23:49:00.864962+0000","last_fullsized":"2024-04-24T23:49:00.864962+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:46.508006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511361+0000","last_change":"2024-04-24T23:48:55.506681+0000","last_active":"2024-04-24T23:49:00.511361+0000","last_peered":"2024-04-24T23:49:00.511361+0000","last_clean":"2024-04-24T23:49:00.511361+0000","last_became_active":"2024-04-24T23:48:55.506393+0000","last_became_peered":"2024-04-24T23:48:55.506393+0000","last_unstale":"2024-04-24T23:49:00.511361+0000","last_undegraded":"2024-04-24T23:49:00.511361+0000","last_fullsized":"2024-04-24T23:49:00.511361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:02:30.176405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863990+0000","last_change":"2024-04-24T23:48:54.499321+0000","last_active":"2024-04-24T23:49:00.863990+0000","last_peered":"2024-04-24T23:49:00.863990+0000","last_clean":"2024-04-24T23:49:00.863990+0000","last_became_active":"2024-04-24T23:48:54.499015+0000","last_became_peered":"2024-04-24T23:48:54.499015+0000","last_unstale":"2024-04-24T23:49:00.863990+0000","last_undegraded":"2024-04-24T23:49:00.863990+0000","last_fullsized":"2024-04-24T23:49:00.863990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863856+0000","last_change":"2024-04-24T23:48:59.522405+0000","last_active":"2024-04-24T23:49:00.863856+0000","last_peered":"2024-04-24T23:49:00.863856+0000","last_clean":"2024-04-24T23:49:00.863856+0000","last_became_active":"2024-04-24T23:48:59.521434+0000","last_became_peered":"2024-04-24T23:48:59.521434+0000","last_unstale":"2024-04-24T23:49:00.863856+0000","last_undegraded":"2024-04-24T23:49:00.863856+0000","last_fullsized":"2024-04-24T23:49:00.863856+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:39.512436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511297+0000","last_change":"2024-04-24T23:48:55.507123+0000","last_active":"2024-04-24T23:49:00.511297+0000","last_peered":"2024-04-24T23:49:00.511297+0000","last_clean":"2024-04-24T23:49:00.511297+0000","last_became_active":"2024-04-24T23:48:55.506838+0000","last_became_peered":"2024-04-24T23:48:55.506838+0000","last_unstale":"2024-04-24T23:49:00.511297+0000","last_undegraded":"2024-04-24T23:49:00.511297+0000","last_fullsized":"2024-04-24T23:49:00.511297+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:25:17.488901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509852+0000","last_change":"2024-04-24T23:48:54.494375+0000","last_active":"2024-04-24T23:49:00.509852+0000","last_peered":"2024-04-24T23:49:00.509852+0000","last_clean":"2024-04-24T23:49:00.509852+0000","last_became_active":"2024-04-24T23:48:54.494080+0000","last_became_peered":"2024-04-24T23:48:54.494080+0000","last_unstale":"2024-04-24T23:49:00.509852+0000","last_undegraded":"2024-04-24T23:49:00.509852+0000","last_fullsized":"2024-04-24T23:49:00.509852+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:04:46.268530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510042+0000","last_change":"2024-04-24T23:48:59.530805+0000","last_active":"2024-04-24T23:49:00.510042+0000","last_peered":"2024-04-24T23:49:00.510042+0000","last_clean":"2024-04-24T23:49:00.510042+0000","last_became_active":"2024-04-24T23:48:59.530126+0000","last_became_peered":"2024-04-24T23:48:59.530126+0000","last_unstale":"2024-04-24T23:49:00.510042+0000","last_undegraded":"2024-04-24T23:49:00.510042+0000","last_fullsized":"2024-04-24T23:49:00.510042+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:50.914824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509597+0000","last_change":"2024-04-24T23:48:55.499881+0000","last_active":"2024-04-24T23:49:00.509597+0000","last_peered":"2024-04-24T23:49:00.509597+0000","last_clean":"2024-04-24T23:49:00.509597+0000","last_became_active":"2024-04-24T23:48:55.499493+0000","last_became_peered":"2024-04-24T23:48:55.499493+0000","last_unstale":"2024-04-24T23:49:00.509597+0000","last_undegraded":"2024-04-24T23:49:00.509597+0000","last_fullsized":"2024-04-24T23:49:00.509597+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:33.222608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863280+0000","last_change":"2024-04-24T23:48:54.493205+0000","last_active":"2024-04-24T23:49:00.863280+0000","last_peered":"2024-04-24T23:49:00.863280+0000","last_clean":"2024-04-24T23:49:00.863280+0000","last_became_active":"2024-04-24T23:48:54.492446+0000","last_became_peered":"2024-04-24T23:48:54.492446+0000","last_unstale":"2024-04-24T23:49:00.863280+0000","last_undegraded":"2024-04-24T23:49:00.863280+0000","last_fullsized":"2024-04-24T23:49:00.863280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:16:25.658826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863935+0000","last_change":"2024-04-24T23:48:59.532383+0000","last_active":"2024-04-24T23:49:00.863935+0000","last_peered":"2024-04-24T23:49:00.863935+0000","last_clean":"2024-04-24T23:49:00.863935+0000","last_became_active":"2024-04-24T23:48:59.529606+0000","last_became_peered":"2024-04-24T23:48:59.529606+0000","last_unstale":"2024-04-24T23:49:00.863935+0000","last_undegraded":"2024-04-24T23:49:00.863935+0000","last_fullsized":"2024-04-24T23:49:00.863935+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:14.315791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864681+0000","last_change":"2024-04-24T23:48:55.499374+0000","last_active":"2024-04-24T23:49:00.864681+0000","last_peered":"2024-04-24T23:49:00.864681+0000","last_clean":"2024-04-24T23:49:00.864681+0000","last_became_active":"2024-04-24T23:48:55.499107+0000","last_became_peered":"2024-04-24T23:48:55.499107+0000","last_unstale":"2024-04-24T23:49:00.864681+0000","last_undegraded":"2024-04-24T23:49:00.864681+0000","last_fullsized":"2024-04-24T23:49:00.864681+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864217+0000","last_change":"2024-04-24T23:48:54.491505+0000","last_active":"2024-04-24T23:49:00.864217+0000","last_peered":"2024-04-24T23:49:00.864217+0000","last_clean":"2024-04-24T23:49:00.864217+0000","last_became_active":"2024-04-24T23:48:54.491230+0000","last_became_peered":"2024-04-24T23:48:54.491230+0000","last_unstale":"2024-04-24T23:49:00.864217+0000","last_undegraded":"2024-04-24T23:49:00.864217+0000","last_fullsized":"2024-04-24T23:49:00.864217+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526800+0000","last_change":"2024-04-24T23:48:59.531330+0000","last_active":"2024-04-24T23:49:00.526800+0000","last_peered":"2024-04-24T23:49:00.526800+0000","last_clean":"2024-04-24T23:49:00.526800+0000","last_became_active":"2024-04-24T23:48:59.530916+0000","last_became_peered":"2024-04-24T23:48:59.530916+0000","last_unstale":"2024-04-24T23:49:00.526800+0000","last_undegraded":"2024-04-24T23:49:00.526800+0000","last_fullsized":"2024-04-24T23:49:00.526800+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:24:43.024671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510174+0000","last_change":"2024-04-24T23:48:55.502184+0000","last_active":"2024-04-24T23:49:00.510174+0000","last_peered":"2024-04-24T23:49:00.510174+0000","last_clean":"2024-04-24T23:49:00.510174+0000","last_became_active":"2024-04-24T23:48:55.501682+0000","last_became_peered":"2024-04-24T23:48:55.501682+0000","last_unstale":"2024-04-24T23:49:00.510174+0000","last_undegraded":"2024-04-24T23:49:00.510174+0000","last_fullsized":"2024-04-24T23:49:00.510174+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:06.267893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864548+0000","last_change":"2024-04-24T23:48:54.499017+0000","last_active":"2024-04-24T23:49:00.864548+0000","last_peered":"2024-04-24T23:49:00.864548+0000","last_clean":"2024-04-24T23:49:00.864548+0000","last_became_active":"2024-04-24T23:48:54.498747+0000","last_became_peered":"2024-04-24T23:48:54.498747+0000","last_unstale":"2024-04-24T23:49:00.864548+0000","last_undegraded":"2024-04-24T23:49:00.864548+0000","last_fullsized":"2024-04-24T23:49:00.864548+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526541+0000","last_change":"2024-04-24T23:48:59.529890+0000","last_active":"2024-04-24T23:49:00.526541+0000","last_peered":"2024-04-24T23:49:00.526541+0000","last_clean":"2024-04-24T23:49:00.526541+0000","last_became_active":"2024-04-24T23:48:59.529534+0000","last_became_peered":"2024-04-24T23:48:59.529534+0000","last_unstale":"2024-04-24T23:49:00.526541+0000","last_undegraded":"2024-04-24T23:49:00.526541+0000","last_fullsized":"2024-04-24T23:49:00.526541+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:38:25.945723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864543+0000","last_change":"2024-04-24T23:48:55.497917+0000","last_active":"2024-04-24T23:49:00.864543+0000","last_peered":"2024-04-24T23:49:00.864543+0000","last_clean":"2024-04-24T23:49:00.864543+0000","last_became_active":"2024-04-24T23:48:55.497158+0000","last_became_peered":"2024-04-24T23:48:55.497158+0000","last_unstale":"2024-04-24T23:49:00.864543+0000","last_undegraded":"2024-04-24T23:49:00.864543+0000","last_fullsized":"2024-04-24T23:49:00.864543+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:51.930595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865010+0000","last_change":"2024-04-24T23:48:54.491006+0000","last_active":"2024-04-24T23:49:00.865010+0000","last_peered":"2024-04-24T23:49:00.865010+0000","last_clean":"2024-04-24T23:49:00.865010+0000","last_became_active":"2024-04-24T23:48:54.489661+0000","last_became_peered":"2024-04-24T23:48:54.489661+0000","last_unstale":"2024-04-24T23:49:00.865010+0000","last_undegraded":"2024-04-24T23:49:00.865010+0000","last_fullsized":"2024-04-24T23:49:00.865010+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:16:25.658826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864966+0000","last_change":"2024-04-24T23:48:59.521225+0000","last_active":"2024-04-24T23:49:00.864966+0000","last_peered":"2024-04-24T23:49:00.864966+0000","last_clean":"2024-04-24T23:49:00.864966+0000","last_became_active":"2024-04-24T23:48:59.520761+0000","last_became_peered":"2024-04-24T23:48:59.520761+0000","last_unstale":"2024-04-24T23:49:00.864966+0000","last_undegraded":"2024-04-24T23:49:00.864966+0000","last_fullsized":"2024-04-24T23:49:00.864966+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:38:25.945723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510500+0000","last_change":"2024-04-24T23:48:55.487564+0000","last_active":"2024-04-24T23:49:00.510500+0000","last_peered":"2024-04-24T23:49:00.510500+0000","last_clean":"2024-04-24T23:49:00.510500+0000","last_became_active":"2024-04-24T23:48:55.487272+0000","last_became_peered":"2024-04-24T23:48:55.487272+0000","last_unstale":"2024-04-24T23:49:00.510500+0000","last_undegraded":"2024-04-24T23:49:00.510500+0000","last_fullsized":"2024-04-24T23:49:00.510500+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:10.235664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":30,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:04.927990+0000","last_change":"2024-04-24T23:48:54.500818+0000","last_active":"2024-04-24T23:49:04.927990+0000","last_peered":"2024-04-24T23:49:04.927990+0000","last_clean":"2024-04-24T23:49:04.927990+0000","last_became_active":"2024-04-24T23:48:54.500560+0000","last_became_peered":"2024-04-24T23:48:54.500560+0000","last_unstale":"2024-04-24T23:49:04.927990+0000","last_undegraded":"2024-04-24T23:49:04.927990+0000","last_fullsized":"2024-04-24T23:49:04.927990+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:24.551434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509881+0000","last_change":"2024-04-24T23:48:59.530292+0000","last_active":"2024-04-24T23:49:00.509881+0000","last_peered":"2024-04-24T23:49:00.509881+0000","last_clean":"2024-04-24T23:49:00.509881+0000","last_became_active":"2024-04-24T23:48:59.529429+0000","last_became_peered":"2024-04-24T23:48:59.529429+0000","last_unstale":"2024-04-24T23:49:00.509881+0000","last_undegraded":"2024-04-24T23:49:00.509881+0000","last_fullsized":"2024-04-24T23:49:00.509881+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:10.292844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510600+0000","last_change":"2024-04-24T23:48:55.502480+0000","last_active":"2024-04-24T23:49:00.510600+0000","last_peered":"2024-04-24T23:49:00.510600+0000","last_clean":"2024-04-24T23:49:00.510600+0000","last_became_active":"2024-04-24T23:48:55.502104+0000","last_became_peered":"2024-04-24T23:48:55.502104+0000","last_unstale":"2024-04-24T23:49:00.510600+0000","last_undegraded":"2024-04-24T23:49:00.510600+0000","last_fullsized":"2024-04-24T23:49:00.510600+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:25.555824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":24,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509534+0000","last_change":"2024-04-24T23:48:54.495473+0000","last_active":"2024-04-24T23:49:00.509534+0000","last_peered":"2024-04-24T23:49:00.509534+0000","last_clean":"2024-04-24T23:49:00.509534+0000","last_became_active":"2024-04-24T23:48:54.495173+0000","last_became_peered":"2024-04-24T23:48:54.495173+0000","last_unstale":"2024-04-24T23:49:00.509534+0000","last_undegraded":"2024-04-24T23:49:00.509534+0000","last_fullsized":"2024-04-24T23:49:00.509534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:24:45.210978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510914+0000","last_change":"2024-04-24T23:48:59.532057+0000","last_active":"2024-04-24T23:49:00.510914+0000","last_peered":"2024-04-24T23:49:00.510914+0000","last_clean":"2024-04-24T23:49:00.510914+0000","last_became_active":"2024-04-24T23:48:59.530378+0000","last_became_peered":"2024-04-24T23:48:59.530378+0000","last_unstale":"2024-04-24T23:49:00.510914+0000","last_undegraded":"2024-04-24T23:49:00.510914+0000","last_fullsized":"2024-04-24T23:49:00.510914+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:51.652134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864215+0000","last_change":"2024-04-24T23:48:55.508829+0000","last_active":"2024-04-24T23:49:00.864215+0000","last_peered":"2024-04-24T23:49:00.864215+0000","last_clean":"2024-04-24T23:49:00.864215+0000","last_became_active":"2024-04-24T23:48:55.508042+0000","last_became_peered":"2024-04-24T23:48:55.508042+0000","last_unstale":"2024-04-24T23:49:00.864215+0000","last_undegraded":"2024-04-24T23:49:00.864215+0000","last_fullsized":"2024-04-24T23:49:00.864215+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:29:20.147177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865051+0000","last_change":"2024-04-24T23:48:54.497886+0000","last_active":"2024-04-24T23:49:00.865051+0000","last_peered":"2024-04-24T23:49:00.865051+0000","last_clean":"2024-04-24T23:49:00.865051+0000","last_became_active":"2024-04-24T23:48:54.497630+0000","last_became_peered":"2024-04-24T23:48:54.497630+0000","last_unstale":"2024-04-24T23:49:00.865051+0000","last_undegraded":"2024-04-24T23:49:00.865051+0000","last_fullsized":"2024-04-24T23:49:00.865051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:31.412349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526184+0000","last_change":"2024-04-24T23:48:59.532080+0000","last_active":"2024-04-24T23:49:00.526184+0000","last_peered":"2024-04-24T23:49:00.526184+0000","last_clean":"2024-04-24T23:49:00.526184+0000","last_became_active":"2024-04-24T23:48:59.531298+0000","last_became_peered":"2024-04-24T23:48:59.531298+0000","last_unstale":"2024-04-24T23:49:00.526184+0000","last_undegraded":"2024-04-24T23:49:00.526184+0000","last_fullsized":"2024-04-24T23:49:00.526184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:33:39.512436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864909+0000","last_change":"2024-04-24T23:48:55.513313+0000","last_active":"2024-04-24T23:49:00.864909+0000","last_peered":"2024-04-24T23:49:00.864909+0000","last_clean":"2024-04-24T23:49:00.864909+0000","last_became_active":"2024-04-24T23:48:55.513021+0000","last_became_peered":"2024-04-24T23:48:55.513021+0000","last_unstale":"2024-04-24T23:49:00.864909+0000","last_undegraded":"2024-04-24T23:49:00.864909+0000","last_fullsized":"2024-04-24T23:49:00.864909+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:04.801548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864612+0000","last_change":"2024-04-24T23:48:54.498900+0000","last_active":"2024-04-24T23:49:00.864612+0000","last_peered":"2024-04-24T23:49:00.864612+0000","last_clean":"2024-04-24T23:49:00.864612+0000","last_became_active":"2024-04-24T23:48:54.498604+0000","last_became_peered":"2024-04-24T23:48:54.498604+0000","last_unstale":"2024-04-24T23:49:00.864612+0000","last_undegraded":"2024-04-24T23:49:00.864612+0000","last_fullsized":"2024-04-24T23:49:00.864612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:15:10.269984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526077+0000","last_change":"2024-04-24T23:48:59.524031+0000","last_active":"2024-04-24T23:49:00.526077+0000","last_peered":"2024-04-24T23:49:00.526077+0000","last_clean":"2024-04-24T23:49:00.526077+0000","last_became_active":"2024-04-24T23:48:59.523585+0000","last_became_peered":"2024-04-24T23:48:59.523585+0000","last_unstale":"2024-04-24T23:49:00.526077+0000","last_undegraded":"2024-04-24T23:49:00.526077+0000","last_fullsized":"2024-04-24T23:49:00.526077+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:33.409892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863757+0000","last_change":"2024-04-24T23:48:59.539246+0000","last_active":"2024-04-24T23:49:00.863757+0000","last_peered":"2024-04-24T23:49:00.863757+0000","last_clean":"2024-04-24T23:49:00.863757+0000","last_became_active":"2024-04-24T23:48:59.538845+0000","last_became_peered":"2024-04-24T23:48:59.538845+0000","last_unstale":"2024-04-24T23:49:00.863757+0000","last_undegraded":"2024-04-24T23:49:00.863757+0000","last_fullsized":"2024-04-24T23:49:00.863757+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:36:37.247560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508807+0000","last_change":"2024-04-24T23:48:55.493299+0000","last_active":"2024-04-24T23:49:00.508807+0000","last_peered":"2024-04-24T23:49:00.508807+0000","last_clean":"2024-04-24T23:49:00.508807+0000","last_became_active":"2024-04-24T23:48:55.492944+0000","last_became_peered":"2024-04-24T23:48:55.492944+0000","last_unstale":"2024-04-24T23:49:00.508807+0000","last_undegraded":"2024-04-24T23:49:00.508807+0000","last_fullsized":"2024-04-24T23:49:00.508807+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:53:38.263114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863928+0000","last_change":"2024-04-24T23:48:54.494727+0000","last_active":"2024-04-24T23:49:00.863928+0000","last_peered":"2024-04-24T23:49:00.863928+0000","last_clean":"2024-04-24T23:49:00.863928+0000","last_became_active":"2024-04-24T23:48:54.494441+0000","last_became_peered":"2024-04-24T23:48:54.494441+0000","last_unstale":"2024-04-24T23:49:00.863928+0000","last_undegraded":"2024-04-24T23:49:00.863928+0000","last_fullsized":"2024-04-24T23:49:00.863928+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:00:40.368478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864551+0000","last_change":"2024-04-24T23:48:59.530686+0000","last_active":"2024-04-24T23:49:00.864551+0000","last_peered":"2024-04-24T23:49:00.864551+0000","last_clean":"2024-04-24T23:49:00.864551+0000","last_became_active":"2024-04-24T23:48:59.529705+0000","last_became_peered":"2024-04-24T23:48:59.529705+0000","last_unstale":"2024-04-24T23:49:00.864551+0000","last_undegraded":"2024-04-24T23:49:00.864551+0000","last_fullsized":"2024-04-24T23:49:00.864551+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:01:10.292844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526143+0000","last_change":"2024-04-24T23:48:55.498040+0000","last_active":"2024-04-24T23:49:00.526143+0000","last_peered":"2024-04-24T23:49:00.526143+0000","last_clean":"2024-04-24T23:49:00.526143+0000","last_became_active":"2024-04-24T23:48:55.497298+0000","last_became_peered":"2024-04-24T23:48:55.497298+0000","last_unstale":"2024-04-24T23:49:00.526143+0000","last_undegraded":"2024-04-24T23:49:00.526143+0000","last_fullsized":"2024-04-24T23:49:00.526143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526688+0000","last_change":"2024-04-24T23:48:54.495330+0000","last_active":"2024-04-24T23:49:00.526688+0000","last_peered":"2024-04-24T23:49:00.526688+0000","last_clean":"2024-04-24T23:49:00.526688+0000","last_became_active":"2024-04-24T23:48:54.495079+0000","last_became_peered":"2024-04-24T23:48:54.495079+0000","last_unstale":"2024-04-24T23:49:00.526688+0000","last_undegraded":"2024-04-24T23:49:00.526688+0000","last_fullsized":"2024-04-24T23:49:00.526688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:16:25.658826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864192+0000","last_change":"2024-04-24T23:48:54.492734+0000","last_active":"2024-04-24T23:49:00.864192+0000","last_peered":"2024-04-24T23:49:00.864192+0000","last_clean":"2024-04-24T23:49:00.864192+0000","last_became_active":"2024-04-24T23:48:54.492467+0000","last_became_peered":"2024-04-24T23:48:54.492467+0000","last_unstale":"2024-04-24T23:49:00.864192+0000","last_undegraded":"2024-04-24T23:49:00.864192+0000","last_fullsized":"2024-04-24T23:49:00.864192+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:03.081371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864956+0000","last_change":"2024-04-24T23:48:55.507847+0000","last_active":"2024-04-24T23:49:00.864956+0000","last_peered":"2024-04-24T23:49:00.864956+0000","last_clean":"2024-04-24T23:49:00.864956+0000","last_became_active":"2024-04-24T23:48:55.507382+0000","last_became_peered":"2024-04-24T23:48:55.507382+0000","last_unstale":"2024-04-24T23:49:00.864956+0000","last_undegraded":"2024-04-24T23:49:00.864956+0000","last_fullsized":"2024-04-24T23:49:00.864956+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863934+0000","last_change":"2024-04-24T23:48:59.521454+0000","last_active":"2024-04-24T23:49:00.863934+0000","last_peered":"2024-04-24T23:49:00.863934+0000","last_clean":"2024-04-24T23:49:00.863934+0000","last_became_active":"2024-04-24T23:48:59.521041+0000","last_became_peered":"2024-04-24T23:48:59.521041+0000","last_unstale":"2024-04-24T23:49:00.863934+0000","last_undegraded":"2024-04-24T23:49:00.863934+0000","last_fullsized":"2024-04-24T23:49:00.863934+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:42.836048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526295+0000","last_change":"2024-04-24T23:48:54.489072+0000","last_active":"2024-04-24T23:49:00.526295+0000","last_peered":"2024-04-24T23:49:00.526295+0000","last_clean":"2024-04-24T23:49:00.526295+0000","last_became_active":"2024-04-24T23:48:54.488735+0000","last_became_peered":"2024-04-24T23:48:54.488735+0000","last_unstale":"2024-04-24T23:49:00.526295+0000","last_undegraded":"2024-04-24T23:49:00.526295+0000","last_fullsized":"2024-04-24T23:49:00.526295+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:53.466592+0000","last_clean_scrub_stamp":"2024-04-24T23:48:53.466592+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:13:26.511856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510340+0000","last_change":"2024-04-24T23:48:55.501095+0000","last_active":"2024-04-24T23:49:00.510340+0000","last_peered":"2024-04-24T23:49:00.510340+0000","last_clean":"2024-04-24T23:49:00.510340+0000","last_became_active":"2024-04-24T23:48:55.500804+0000","last_became_peered":"2024-04-24T23:48:55.500804+0000","last_unstale":"2024-04-24T23:49:00.510340+0000","last_undegraded":"2024-04-24T23:49:00.510340+0000","last_fullsized":"2024-04-24T23:49:00.510340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:15:11.274374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526847+0000","last_change":"2024-04-24T23:48:59.532387+0000","last_active":"2024-04-24T23:49:00.526847+0000","last_peered":"2024-04-24T23:49:00.526847+0000","last_clean":"2024-04-24T23:49:00.526847+0000","last_became_active":"2024-04-24T23:48:59.531805+0000","last_became_peered":"2024-04-24T23:48:59.531805+0000","last_unstale":"2024-04-24T23:49:00.526847+0000","last_undegraded":"2024-04-24T23:49:00.526847+0000","last_fullsized":"2024-04-24T23:49:00.526847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:58.495933+0000","last_clean_scrub_stamp":"2024-04-24T23:48:58.495933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:10.597258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863811+0000","last_change":"2024-04-24T23:48:55.502131+0000","last_active":"2024-04-24T23:49:00.863811+0000","last_peered":"2024-04-24T23:49:00.863811+0000","last_clean":"2024-04-24T23:49:00.863811+0000","last_became_active":"2024-04-24T23:48:55.501871+0000","last_became_peered":"2024-04-24T23:48:55.501871+0000","last_unstale":"2024-04-24T23:49:00.863811+0000","last_undegraded":"2024-04-24T23:49:00.863811+0000","last_fullsized":"2024-04-24T23:49:00.863811+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863465+0000","last_change":"2024-04-24T23:48:55.503813+0000","last_active":"2024-04-24T23:49:00.863465+0000","last_peered":"2024-04-24T23:49:00.863465+0000","last_clean":"2024-04-24T23:49:00.863465+0000","last_became_active":"2024-04-24T23:48:55.503517+0000","last_became_peered":"2024-04-24T23:48:55.503517+0000","last_unstale":"2024-04-24T23:49:00.863465+0000","last_undegraded":"2024-04-24T23:49:00.863465+0000","last_fullsized":"2024-04-24T23:49:00.863465+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:33:25.555824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864441+0000","last_change":"2024-04-24T23:48:55.507595+0000","last_active":"2024-04-24T23:49:00.864441+0000","last_peered":"2024-04-24T23:49:00.864441+0000","last_clean":"2024-04-24T23:49:00.864441+0000","last_became_active":"2024-04-24T23:48:55.507307+0000","last_became_peered":"2024-04-24T23:48:55.507307+0000","last_unstale":"2024-04-24T23:49:00.864441+0000","last_undegraded":"2024-04-24T23:49:00.864441+0000","last_fullsized":"2024-04-24T23:49:00.864441+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:10.290840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864867+0000","last_change":"2024-04-24T23:48:55.498028+0000","last_active":"2024-04-24T23:49:00.864867+0000","last_peered":"2024-04-24T23:49:00.864867+0000","last_clean":"2024-04-24T23:49:00.864867+0000","last_became_active":"2024-04-24T23:48:55.496715+0000","last_became_peered":"2024-04-24T23:48:55.496715+0000","last_unstale":"2024-04-24T23:49:00.864867+0000","last_undegraded":"2024-04-24T23:49:00.864867+0000","last_fullsized":"2024-04-24T23:49:00.864867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526777+0000","last_change":"2024-04-24T23:48:55.505719+0000","last_active":"2024-04-24T23:49:00.526777+0000","last_peered":"2024-04-24T23:49:00.526777+0000","last_clean":"2024-04-24T23:49:00.526777+0000","last_became_active":"2024-04-24T23:48:55.505418+0000","last_became_peered":"2024-04-24T23:48:55.505418+0000","last_unstale":"2024-04-24T23:49:00.526777+0000","last_undegraded":"2024-04-24T23:49:00.526777+0000","last_fullsized":"2024-04-24T23:49:00.526777+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509327+0000","last_change":"2024-04-24T23:48:55.492403+0000","last_active":"2024-04-24T23:49:00.509327+0000","last_peered":"2024-04-24T23:49:00.509327+0000","last_clean":"2024-04-24T23:49:00.509327+0000","last_became_active":"2024-04-24T23:48:55.492112+0000","last_became_peered":"2024-04-24T23:48:55.492112+0000","last_unstale":"2024-04-24T23:49:00.509327+0000","last_undegraded":"2024-04-24T23:49:00.509327+0000","last_fullsized":"2024-04-24T23:49:00.509327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:01:34.545127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864731+0000","last_change":"2024-04-24T23:48:55.488480+0000","last_active":"2024-04-24T23:49:00.864731+0000","last_peered":"2024-04-24T23:49:00.864731+0000","last_clean":"2024-04-24T23:49:00.864731+0000","last_became_active":"2024-04-24T23:48:55.488134+0000","last_became_peered":"2024-04-24T23:48:55.488134+0000","last_unstale":"2024-04-24T23:49:00.864731+0000","last_undegraded":"2024-04-24T23:49:00.864731+0000","last_fullsized":"2024-04-24T23:49:00.864731+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:55.758743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526760+0000","last_change":"2024-04-24T23:48:55.507589+0000","last_active":"2024-04-24T23:49:00.526760+0000","last_peered":"2024-04-24T23:49:00.526760+0000","last_clean":"2024-04-24T23:49:00.526760+0000","last_became_active":"2024-04-24T23:48:55.507281+0000","last_became_peered":"2024-04-24T23:48:55.507281+0000","last_unstale":"2024-04-24T23:49:00.526760+0000","last_undegraded":"2024-04-24T23:49:00.526760+0000","last_fullsized":"2024-04-24T23:49:00.526760+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:01:51.930595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863920+0000","last_change":"2024-04-24T23:48:55.498917+0000","last_active":"2024-04-24T23:49:00.863920+0000","last_peered":"2024-04-24T23:49:00.863920+0000","last_clean":"2024-04-24T23:49:00.863920+0000","last_became_active":"2024-04-24T23:48:55.498627+0000","last_became_peered":"2024-04-24T23:48:55.498627+0000","last_unstale":"2024-04-24T23:49:00.863920+0000","last_undegraded":"2024-04-24T23:49:00.863920+0000","last_fullsized":"2024-04-24T23:49:00.863920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:55.758743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526124+0000","last_change":"2024-04-24T23:48:55.507449+0000","last_active":"2024-04-24T23:49:00.526124+0000","last_peered":"2024-04-24T23:49:00.526124+0000","last_clean":"2024-04-24T23:49:00.526124+0000","last_became_active":"2024-04-24T23:48:55.507151+0000","last_became_peered":"2024-04-24T23:48:55.507151+0000","last_unstale":"2024-04-24T23:49:00.526124+0000","last_undegraded":"2024-04-24T23:49:00.526124+0000","last_fullsized":"2024-04-24T23:49:00.526124+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:46.889872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864334+0000","last_change":"2024-04-24T23:48:55.511263+0000","last_active":"2024-04-24T23:49:00.864334+0000","last_peered":"2024-04-24T23:49:00.864334+0000","last_clean":"2024-04-24T23:49:00.864334+0000","last_became_active":"2024-04-24T23:48:55.510422+0000","last_became_peered":"2024-04-24T23:48:55.510422+0000","last_unstale":"2024-04-24T23:49:00.864334+0000","last_undegraded":"2024-04-24T23:49:00.864334+0000","last_fullsized":"2024-04-24T23:49:00.864334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:47.627183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864769+0000","last_change":"2024-04-24T23:48:55.508355+0000","last_active":"2024-04-24T23:49:00.864769+0000","last_peered":"2024-04-24T23:49:00.864769+0000","last_clean":"2024-04-24T23:49:00.864769+0000","last_became_active":"2024-04-24T23:48:55.506836+0000","last_became_peered":"2024-04-24T23:48:55.506836+0000","last_unstale":"2024-04-24T23:49:00.864769+0000","last_undegraded":"2024-04-24T23:49:00.864769+0000","last_fullsized":"2024-04-24T23:49:00.864769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:53:38.263114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864802+0000","last_change":"2024-04-24T23:48:55.507774+0000","last_active":"2024-04-24T23:49:00.864802+0000","last_peered":"2024-04-24T23:49:00.864802+0000","last_clean":"2024-04-24T23:49:00.864802+0000","last_became_active":"2024-04-24T23:48:55.506816+0000","last_became_peered":"2024-04-24T23:48:55.506816+0000","last_unstale":"2024-04-24T23:49:00.864802+0000","last_undegraded":"2024-04-24T23:49:00.864802+0000","last_fullsized":"2024-04-24T23:49:00.864802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510851+0000","last_change":"2024-04-24T23:48:55.492334+0000","last_active":"2024-04-24T23:49:00.510851+0000","last_peered":"2024-04-24T23:49:00.510851+0000","last_clean":"2024-04-24T23:49:00.510851+0000","last_became_active":"2024-04-24T23:48:55.492065+0000","last_became_peered":"2024-04-24T23:48:55.492065+0000","last_unstale":"2024-04-24T23:49:00.510851+0000","last_undegraded":"2024-04-24T23:49:00.510851+0000","last_fullsized":"2024-04-24T23:49:00.510851+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:08:42.483054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863835+0000","last_change":"2024-04-24T23:48:55.495084+0000","last_active":"2024-04-24T23:49:00.863835+0000","last_peered":"2024-04-24T23:49:00.863835+0000","last_clean":"2024-04-24T23:49:00.863835+0000","last_became_active":"2024-04-24T23:48:55.494776+0000","last_became_peered":"2024-04-24T23:48:55.494776+0000","last_unstale":"2024-04-24T23:49:00.863835+0000","last_undegraded":"2024-04-24T23:49:00.863835+0000","last_fullsized":"2024-04-24T23:49:00.863835+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:25:32.416739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.865234+0000","last_change":"2024-04-24T23:48:55.511141+0000","last_active":"2024-04-24T23:49:00.865234+0000","last_peered":"2024-04-24T23:49:00.865234+0000","last_clean":"2024-04-24T23:49:00.865234+0000","last_became_active":"2024-04-24T23:48:55.510836+0000","last_became_peered":"2024-04-24T23:48:55.510836+0000","last_unstale":"2024-04-24T23:49:00.865234+0000","last_undegraded":"2024-04-24T23:49:00.865234+0000","last_fullsized":"2024-04-24T23:49:00.865234+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:18.684092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.863787+0000","last_change":"2024-04-24T23:48:55.508632+0000","last_active":"2024-04-24T23:49:00.863787+0000","last_peered":"2024-04-24T23:49:00.863787+0000","last_clean":"2024-04-24T23:49:00.863787+0000","last_became_active":"2024-04-24T23:48:55.507448+0000","last_became_peered":"2024-04-24T23:48:55.507448+0000","last_unstale":"2024-04-24T23:49:00.863787+0000","last_undegraded":"2024-04-24T23:49:00.863787+0000","last_fullsized":"2024-04-24T23:49:00.863787+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:46.889872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.508815+0000","last_change":"2024-04-24T23:48:55.508181+0000","last_active":"2024-04-24T23:49:00.508815+0000","last_peered":"2024-04-24T23:49:00.508815+0000","last_clean":"2024-04-24T23:49:00.508815+0000","last_became_active":"2024-04-24T23:48:55.507877+0000","last_became_peered":"2024-04-24T23:48:55.507877+0000","last_unstale":"2024-04-24T23:49:00.508815+0000","last_undegraded":"2024-04-24T23:49:00.508815+0000","last_fullsized":"2024-04-24T23:49:00.508815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:19:47.070959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864457+0000","last_change":"2024-04-24T23:48:55.497468+0000","last_active":"2024-04-24T23:49:00.864457+0000","last_peered":"2024-04-24T23:49:00.864457+0000","last_clean":"2024-04-24T23:49:00.864457+0000","last_became_active":"2024-04-24T23:48:55.496754+0000","last_became_peered":"2024-04-24T23:48:55.496754+0000","last_unstale":"2024-04-24T23:49:00.864457+0000","last_undegraded":"2024-04-24T23:49:00.864457+0000","last_fullsized":"2024-04-24T23:49:00.864457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:26:58.209422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.526144+0000","last_change":"2024-04-24T23:48:55.492306+0000","last_active":"2024-04-24T23:49:00.526144+0000","last_peered":"2024-04-24T23:49:00.526144+0000","last_clean":"2024-04-24T23:49:00.526144+0000","last_became_active":"2024-04-24T23:48:55.492008+0000","last_became_peered":"2024-04-24T23:48:55.492008+0000","last_unstale":"2024-04-24T23:49:00.526144+0000","last_undegraded":"2024-04-24T23:49:00.526144+0000","last_fullsized":"2024-04-24T23:49:00.526144+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:18:55.758743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.510496+0000","last_change":"2024-04-24T23:48:55.501767+0000","last_active":"2024-04-24T23:49:00.510496+0000","last_peered":"2024-04-24T23:49:00.510496+0000","last_clean":"2024-04-24T23:49:00.510496+0000","last_became_active":"2024-04-24T23:48:55.500776+0000","last_became_peered":"2024-04-24T23:48:55.500776+0000","last_unstale":"2024-04-24T23:49:00.510496+0000","last_undegraded":"2024-04-24T23:49:00.510496+0000","last_fullsized":"2024-04-24T23:49:00.510496+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:53:04.085761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864571+0000","last_change":"2024-04-24T23:48:55.511220+0000","last_active":"2024-04-24T23:49:00.864571+0000","last_peered":"2024-04-24T23:49:00.864571+0000","last_clean":"2024-04-24T23:49:00.864571+0000","last_became_active":"2024-04-24T23:48:55.510937+0000","last_became_peered":"2024-04-24T23:48:55.510937+0000","last_unstale":"2024-04-24T23:49:00.864571+0000","last_undegraded":"2024-04-24T23:49:00.864571+0000","last_fullsized":"2024-04-24T23:49:00.864571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864622+0000","last_change":"2024-04-24T23:48:55.503843+0000","last_active":"2024-04-24T23:49:00.864622+0000","last_peered":"2024-04-24T23:49:00.864622+0000","last_clean":"2024-04-24T23:49:00.864622+0000","last_became_active":"2024-04-24T23:48:55.503569+0000","last_became_peered":"2024-04-24T23:48:55.503569+0000","last_unstale":"2024-04-24T23:49:00.864622+0000","last_undegraded":"2024-04-24T23:49:00.864622+0000","last_fullsized":"2024-04-24T23:49:00.864622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:40:46.889872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511357+0000","last_change":"2024-04-24T23:48:55.509090+0000","last_active":"2024-04-24T23:49:00.511357+0000","last_peered":"2024-04-24T23:49:00.511357+0000","last_clean":"2024-04-24T23:49:00.511357+0000","last_became_active":"2024-04-24T23:48:55.508730+0000","last_became_peered":"2024-04-24T23:48:55.508730+0000","last_unstale":"2024-04-24T23:49:00.511357+0000","last_undegraded":"2024-04-24T23:49:00.511357+0000","last_fullsized":"2024-04-24T23:49:00.511357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:07:18.684092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.525876+0000","last_change":"2024-04-24T23:48:55.498351+0000","last_active":"2024-04-24T23:49:00.525876+0000","last_peered":"2024-04-24T23:49:00.525876+0000","last_clean":"2024-04-24T23:49:00.525876+0000","last_became_active":"2024-04-24T23:48:55.497694+0000","last_became_peered":"2024-04-24T23:48:55.497694+0000","last_unstale":"2024-04-24T23:49:00.525876+0000","last_undegraded":"2024-04-24T23:49:00.525876+0000","last_fullsized":"2024-04-24T23:49:00.525876+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:26:20.025727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509595+0000","last_change":"2024-04-24T23:48:55.501677+0000","last_active":"2024-04-24T23:49:00.509595+0000","last_peered":"2024-04-24T23:49:00.509595+0000","last_clean":"2024-04-24T23:49:00.509595+0000","last_became_active":"2024-04-24T23:48:55.501359+0000","last_became_peered":"2024-04-24T23:48:55.501359+0000","last_unstale":"2024-04-24T23:49:00.509595+0000","last_undegraded":"2024-04-24T23:49:00.509595+0000","last_fullsized":"2024-04-24T23:49:00.509595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:06:47.627183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864613+0000","last_change":"2024-04-24T23:48:55.511089+0000","last_active":"2024-04-24T23:49:00.864613+0000","last_peered":"2024-04-24T23:49:00.864613+0000","last_clean":"2024-04-24T23:49:00.864613+0000","last_became_active":"2024-04-24T23:48:55.510818+0000","last_became_peered":"2024-04-24T23:48:55.510818+0000","last_unstale":"2024-04-24T23:49:00.864613+0000","last_undegraded":"2024-04-24T23:49:00.864613+0000","last_fullsized":"2024-04-24T23:49:00.864613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:03:28.472120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509524+0000","last_change":"2024-04-24T23:48:55.492973+0000","last_active":"2024-04-24T23:49:00.509524+0000","last_peered":"2024-04-24T23:49:00.509524+0000","last_clean":"2024-04-24T23:49:00.509524+0000","last_became_active":"2024-04-24T23:48:55.492713+0000","last_became_peered":"2024-04-24T23:48:55.492713+0000","last_unstale":"2024-04-24T23:49:00.509524+0000","last_undegraded":"2024-04-24T23:49:00.509524+0000","last_fullsized":"2024-04-24T23:49:00.509524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:41:10.290840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864561+0000","last_change":"2024-04-24T23:48:55.508975+0000","last_active":"2024-04-24T23:49:00.864561+0000","last_peered":"2024-04-24T23:49:00.864561+0000","last_clean":"2024-04-24T23:49:00.864561+0000","last_became_active":"2024-04-24T23:48:55.507970+0000","last_became_peered":"2024-04-24T23:48:55.507970+0000","last_unstale":"2024-04-24T23:49:00.864561+0000","last_undegraded":"2024-04-24T23:49:00.864561+0000","last_fullsized":"2024-04-24T23:49:00.864561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:02:10.235664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.864614+0000","last_change":"2024-04-24T23:48:55.500274+0000","last_active":"2024-04-24T23:49:00.864614+0000","last_peered":"2024-04-24T23:49:00.864614+0000","last_clean":"2024-04-24T23:49:00.864614+0000","last_became_active":"2024-04-24T23:48:55.500004+0000","last_became_peered":"2024-04-24T23:48:55.500004+0000","last_unstale":"2024-04-24T23:49:00.864614+0000","last_undegraded":"2024-04-24T23:49:00.864614+0000","last_fullsized":"2024-04-24T23:49:00.864614+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:02:47.989664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.509710+0000","last_change":"2024-04-24T23:48:55.499555+0000","last_active":"2024-04-24T23:49:00.509710+0000","last_peered":"2024-04-24T23:49:00.509710+0000","last_clean":"2024-04-24T23:49:00.509710+0000","last_became_active":"2024-04-24T23:48:55.499163+0000","last_became_peered":"2024-04-24T23:48:55.499163+0000","last_unstale":"2024-04-24T23:49:00.509710+0000","last_undegraded":"2024-04-24T23:49:00.509710+0000","last_fullsized":"2024-04-24T23:49:00.509710+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:29:43.827150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-04-24T23:49:00.511249+0000","last_change":"2024-04-24T23:48:55.493200+0000","last_active":"2024-04-24T23:49:00.511249+0000","last_peered":"2024-04-24T23:49:00.511249+0000","last_clean":"2024-04-24T23:49:00.511249+0000","last_became_active":"2024-04-24T23:48:55.491655+0000","last_became_peered":"2024-04-24T23:48:55.491655+0000","last_unstale":"2024-04-24T23:49:00.511249+0000","last_undegraded":"2024-04-24T23:49:00.511249+0000","last_fullsized":"2024-04-24T23:49:00.511249+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:54.470982+0000","last_clean_scrub_stamp":"2024-04-24T23:48:54.470982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:59:04.801548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574859,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":864256,"data_stored":695877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31452,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340388,"statfs":{"total":96636764160,"available":96604557312,"internally_reserved":0,"allocated":270336,"data_stored":105017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574859,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31460,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340380,"statfs":{"total":96636764160,"available":96604549120,"internally_reserved":0,"allocated":278528,"data_stored":105053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":278528,"data_stored":108769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574860,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":104927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":274432,"data_stored":105087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574859,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32036,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339804,"statfs":{"total":96636764160,"available":96603959296,"internally_reserved":0,"allocated":872448,"data_stored":695933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:49:25.582 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T23:49:25.582 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-24T23:49:25.582 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-04-24T23:49:25.582 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-04-24T23:49:25.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.717+0000 7fa31a493640 1 Processor -- start 2024-04-24T23:49:25.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.717+0000 7fa31a493640 1 -- start start 2024-04-24T23:49:25.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.718+0000 7fa31a493640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa314071a00 0x7fa314071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.718+0000 7fa31a493640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa314072320 con 0x7fa3140ce7c0 2024-04-24T23:49:25.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.718+0000 7fa31a493640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3140cf3a0 con 0x7fa314144740 2024-04-24T23:49:25.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.718+0000 7fa31a493640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa31415c1f0 con 0x7fa314071a00 2024-04-24T23:49:25.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.718+0000 7fa318a30640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa3140ce7c0 0x7fa3140cebf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50754/0 (socket says 172.21.15.16:50754) 2024-04-24T23:49:25.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.718+0000 7fa318a30640 1 -- 172.21.15.16:0/503554571 learned_addr learned my addr 172.21.15.16:0/503554571 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:25.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.718+0000 7fa313fff640 1 --2- 172.21.15.16:0/503554571 >> v2:172.21.15.156:3300/0 conn(0x7fa314071a00 0x7fa314071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.718+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 208007621 0 0) 0x7fa314072320 con 0x7fa3140ce7c0 2024-04-24T23:49:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.718+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa300003520 con 0x7fa3140ce7c0 2024-04-24T23:49:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.718+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2636906610 0 0) 0x7fa3140cf3a0 con 0x7fa314144740 2024-04-24T23:49:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa314072320 con 0x7fa314144740 2024-04-24T23:49:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 326441086 0 0) 0x7fa300003520 con 0x7fa3140ce7c0 2024-04-24T23:49:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 >> v2:172.21.15.156:3300/0 conn(0x7fa314071a00 msgr2=0x7fa314071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa312ffd640 1 --2- 172.21.15.16:0/503554571 >> v2:172.21.15.156:3300/0 conn(0x7fa314071a00 0x7fa314071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 >> v1:172.21.15.156:6789/0 conn(0x7fa314144740 legacy=0x7fa31415c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3140cc8d0 con 0x7fa3140ce7c0 2024-04-24T23:49:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7fa30c001970 con 0x7fa3140ce7c0 2024-04-24T23:49:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 >> v1:172.21.15.16:6789/0 conn(0x7fa3140ce7c0 legacy=0x7fa3140cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa312ffd640 1 --2- 172.21.15.16:0/503554571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa300003b90 0x7fa300003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa312ffd640 1 --2- 172.21.15.16:0/503554571 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa300004800 0x7fa300004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa312ffd640 1 --2- 172.21.15.16:0/503554571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3000051c0 0x7fa30000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa30000c090 con 0x7fa300003b90 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa30000c210 con 0x7fa3000051c0 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fa30000c410 con 0x7fa300004800 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa313fff640 1 --2- 172.21.15.16:0/503554571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa300003b90 0x7fa300003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa3137fe640 1 --2- 172.21.15.16:0/503554571 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa300004800 0x7fa300004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa318a30640 1 --2- 172.21.15.16:0/503554571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3000051c0 0x7fa30000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa313fff640 1 -- 172.21.15.16:0/503554571 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa300004800 msgr2=0x7fa300004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa313fff640 1 --2- 172.21.15.16:0/503554571 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa300004800 0x7fa300004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa313fff640 1 -- 172.21.15.16:0/503554571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3000051c0 msgr2=0x7fa30000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa313fff640 1 --2- 172.21.15.16:0/503554571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3000051c0 0x7fa30000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa313fff640 1 -- 172.21.15.16:0/503554571 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa300003520 con 0x7fa300003b90 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa3137fe640 1 --2- 172.21.15.16:0/503554571 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa300004800 0x7fa300004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa318a30640 1 --2- 172.21.15.16:0/503554571 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa3000051c0 0x7fa30000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.719+0000 7fa318a30640 1 -- 172.21.15.16:0/503554571 reap_dead start 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.720+0000 7fa313fff640 1 --2- 172.21.15.16:0/503554571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa300003b90 0x7fa300003f90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa304009ac0 tx=0x7fa300005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b071a49aa8176656 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:25.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.720+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa304036070 con 0x7fa300003b90 2024-04-24T23:49:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.720+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa304031400 con 0x7fa300003b90 2024-04-24T23:49:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.720+0000 7fa312ffd640 1 -- 172.21.15.16:0/503554571 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa304009690 con 0x7fa300003b90 2024-04-24T23:49:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.720+0000 7fa31a493640 1 -- 172.21.15.16:0/503554571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa300003b90 msgr2=0x7fa300003f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.720+0000 7fa31a493640 1 --2- 172.21.15.16:0/503554571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa300003b90 0x7fa300003f90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fa304009ac0 tx=0x7fa300005830 comp rx=0 tx=0).stop 2024-04-24T23:49:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.720+0000 7fa31a493640 1 -- 172.21.15.16:0/503554571 shutdown_connections 2024-04-24T23:49:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.720+0000 7fa31a493640 1 --2- 172.21.15.16:0/503554571 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa300003b90 0x7fa300003f90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.720+0000 7fa31a493640 1 -- 172.21.15.16:0/503554571 >> 172.21.15.16:0/503554571 conn(0x7fa3140b2000 msgr2=0x7fa3140cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.720+0000 7fa31a493640 1 -- 172.21.15.16:0/503554571 shutdown_connections 2024-04-24T23:49:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.720+0000 7fa31a493640 1 -- 172.21.15.16:0/503554571 wait complete. 2024-04-24T23:49:25.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.722+0000 7fa31a493640 1 Processor -- start 2024-04-24T23:49:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.722+0000 7fa31a493640 1 -- start start 2024-04-24T23:49:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.722+0000 7fa31a493640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa300003b90 0x7fa3140b0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.722+0000 7fa31a493640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa31415a140 0x7fa31415a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.722+0000 7fa31a493640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa31413d7e0 0x7fa31413dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.722+0000 7fa31a493640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa3140cd170 con 0x7fa300003b90 2024-04-24T23:49:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.722+0000 7fa31a493640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa3140cce90 con 0x7fa31415a140 2024-04-24T23:49:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa31a493640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fa314071de0 con 0x7fa31413d7e0 2024-04-24T23:49:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa313fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa300003b90 0x7fa3140b0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa313fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa300003b90 0x7fa3140b0930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48756/0 (socket says 172.21.15.16:48756) 2024-04-24T23:49:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa313fff640 1 -- 172.21.15.16:0/231136705 learned_addr learned my addr 172.21.15.16:0/231136705 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa318a30640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa31413d7e0 0x7fa31413dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa3137fe640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa31415a140 0x7fa31415a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa313fff640 1 -- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa31413d7e0 msgr2=0x7fa31413dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa313fff640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa31413d7e0 0x7fa31413dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa313fff640 1 -- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa31415a140 msgr2=0x7fa31415a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa313fff640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa31415a140 0x7fa31415a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa313fff640 1 -- 172.21.15.16:0/231136705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa31413e550 con 0x7fa300003b90 2024-04-24T23:49:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa318a30640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa31413d7e0 0x7fa31413dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa313fff640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa300003b90 0x7fa3140b0930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa304009be0 tx=0x7fa304004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa310ff9640 1 -- 172.21.15.16:0/231136705 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30403f020 con 0x7fa300003b90 2024-04-24T23:49:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa310ff9640 1 -- 172.21.15.16:0/231136705 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa304037040 con 0x7fa300003b90 2024-04-24T23:49:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa310ff9640 1 -- 172.21.15.16:0/231136705 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa304036070 con 0x7fa300003b90 2024-04-24T23:49:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa31a493640 1 -- 172.21.15.16:0/231136705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa31413e830 con 0x7fa300003b90 2024-04-24T23:49:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.723+0000 7fa31a493640 1 -- 172.21.15.16:0/231136705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3141e3880 con 0x7fa300003b90 2024-04-24T23:49:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.724+0000 7fa31a493640 1 -- 172.21.15.16:0/231136705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3140731f0 con 0x7fa300003b90 2024-04-24T23:49:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.725+0000 7fa310ff9640 1 -- 172.21.15.16:0/231136705 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa304042020 con 0x7fa300003b90 2024-04-24T23:49:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.725+0000 7fa310ff9640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa2f406be30 0x7fa2f406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.725+0000 7fa310ff9640 1 -- 172.21.15.16:0/231136705 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fa3040aaef0 con 0x7fa300003b90 2024-04-24T23:49:25.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.725+0000 7fa3137fe640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa2f406be30 0x7fa2f406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.726+0000 7fa3137fe640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa2f406be30 0x7fa2f406e2f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc007fd0 tx=0x7fa2fc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:25.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.730+0000 7fa310ff9640 1 -- 172.21.15.16:0/231136705 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa304049070 con 0x7fa300003b90 2024-04-24T23:49:25.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.949+0000 7fa31a493640 1 -- 172.21.15.16:0/231136705 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fa31404f5e0 con 0x7fa300003b90 2024-04-24T23:49:25.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.949+0000 7fa310ff9640 1 -- 172.21.15.16:0/231136705 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fa3040795f0 con 0x7fa300003b90 2024-04-24T23:49:25.949 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:25.950 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T23:49:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.951+0000 7fa31a493640 1 -- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa2f406be30 msgr2=0x7fa2f406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.951+0000 7fa31a493640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa2f406be30 0x7fa2f406e2f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc007fd0 tx=0x7fa2fc009040 comp rx=0 tx=0).stop 2024-04-24T23:49:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.951+0000 7fa31a493640 1 -- 172.21.15.16:0/231136705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa300003b90 msgr2=0x7fa3140b0930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.951+0000 7fa31a493640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa300003b90 0x7fa3140b0930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa304009be0 tx=0x7fa304004730 comp rx=0 tx=0).stop 2024-04-24T23:49:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.951+0000 7fa31a493640 1 -- 172.21.15.16:0/231136705 shutdown_connections 2024-04-24T23:49:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.951+0000 7fa31a493640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa2f406be30 0x7fa2f406e2f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.951+0000 7fa31a493640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa31413d7e0 0x7fa31413dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.951+0000 7fa31a493640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa31415a140 0x7fa31415a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.951+0000 7fa31a493640 1 --2- 172.21.15.16:0/231136705 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa300003b90 0x7fa3140b0930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:25.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.951+0000 7fa31a493640 1 -- 172.21.15.16:0/231136705 >> 172.21.15.16:0/231136705 conn(0x7fa3140b2000 msgr2=0x7fa31415d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:25.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.951+0000 7fa31a493640 1 -- 172.21.15.16:0/231136705 shutdown_connections 2024-04-24T23:49:25.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:25.951+0000 7fa31a493640 1 -- 172.21.15.16:0/231136705 wait complete. 2024-04-24T23:49:25.962 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-04-24T23:49:25.962 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:26.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.063+0000 7fc448695640 1 Processor -- start 2024-04-24T23:49:26.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.063+0000 7fc448695640 1 -- start start 2024-04-24T23:49:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.064+0000 7fc448695640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4401457b0 0x7fc4400ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.064+0000 7fc448695640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc44015f990 0x7fc4400cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.064+0000 7fc448695640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc44015fe90 con 0x7fc440144b20 2024-04-24T23:49:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.064+0000 7fc448695640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fc44006ea50 con 0x7fc44015f990 2024-04-24T23:49:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.064+0000 7fc448695640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fc440160170 con 0x7fc4401457b0 2024-04-24T23:49:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.064+0000 7fc445c30640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4401457b0 0x7fc4400ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.064+0000 7fc446c32640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc44015f990 0x7fc4400cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.064+0000 7fc445c30640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4401457b0 0x7fc4400ce680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:41782/0 (socket says 172.21.15.16:41782) 2024-04-24T23:49:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.064+0000 7fc445c30640 1 -- 172.21.15.16:0/1011298838 learned_addr learned my addr 172.21.15.16:0/1011298838 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.064+0000 7fc446c32640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc44015f990 0x7fc4400cebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:35160/0 (socket says 172.21.15.16:35160) 2024-04-24T23:49:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.064+0000 7fc446431640 1 --1- 172.21.15.16:0/1011298838 >> v1:172.21.15.156:6789/0 conn(0x7fc440144b20 0x7fc440144f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:41380/0 (socket says 172.21.15.16:41380) 2024-04-24T23:49:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.065+0000 7fc445c30640 1 -- 172.21.15.16:0/1011298838 >> v2:172.21.15.156:3300/0 conn(0x7fc44015f990 msgr2=0x7fc4400cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.065+0000 7fc445c30640 1 --2- 172.21.15.16:0/1011298838 >> v2:172.21.15.156:3300/0 conn(0x7fc44015f990 0x7fc4400cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.065+0000 7fc445c30640 1 -- 172.21.15.16:0/1011298838 >> v1:172.21.15.156:6789/0 conn(0x7fc440144b20 legacy=0x7fc440144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.065+0000 7fc445c30640 1 -- 172.21.15.16:0/1011298838 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc440160450 con 0x7fc4401457b0 2024-04-24T23:49:26.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.065+0000 7fc445c30640 1 --2- 172.21.15.16:0/1011298838 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4401457b0 0x7fc4400ce680 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc434009870 tx=0x7fc43402a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a6b16516f618b67 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:26.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.066+0000 7fc44542f640 1 -- 172.21.15.16:0/1011298838 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc434037070 con 0x7fc4401457b0 2024-04-24T23:49:26.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.066+0000 7fc44542f640 1 -- 172.21.15.16:0/1011298838 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc434032440 con 0x7fc4401457b0 2024-04-24T23:49:26.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.066+0000 7fc44542f640 1 -- 172.21.15.16:0/1011298838 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc434032810 con 0x7fc4401457b0 2024-04-24T23:49:26.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.066+0000 7fc448695640 1 -- 172.21.15.16:0/1011298838 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4401457b0 msgr2=0x7fc4400ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.066+0000 7fc448695640 1 --2- 172.21.15.16:0/1011298838 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4401457b0 0x7fc4400ce680 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc434009870 tx=0x7fc43402a1c0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.066+0000 7fc448695640 1 -- 172.21.15.16:0/1011298838 shutdown_connections 2024-04-24T23:49:26.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.066+0000 7fc448695640 1 --2- 172.21.15.16:0/1011298838 >> v2:172.21.15.156:3300/0 conn(0x7fc44015f990 0x7fc4400cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.066+0000 7fc448695640 1 --2- 172.21.15.16:0/1011298838 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4401457b0 0x7fc4400ce680 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.066+0000 7fc448695640 1 -- 172.21.15.16:0/1011298838 >> 172.21.15.16:0/1011298838 conn(0x7fc4400b2000 msgr2=0x7fc4400cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:26.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.066+0000 7fc448695640 1 -- 172.21.15.16:0/1011298838 shutdown_connections 2024-04-24T23:49:26.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.066+0000 7fc448695640 1 -- 172.21.15.16:0/1011298838 wait complete. 2024-04-24T23:49:26.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.067+0000 7fc448695640 1 Processor -- start 2024-04-24T23:49:26.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.067+0000 7fc448695640 1 -- start start 2024-04-24T23:49:26.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.068+0000 7fc448695640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc440144b20 0x7fc440159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.068+0000 7fc448695640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4401457b0 0x7fc44015e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.068+0000 7fc448695640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc44015f990 0x7fc4401f1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.068+0000 7fc448695640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4400b3390 con 0x7fc440144b20 2024-04-24T23:49:26.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.068+0000 7fc448695640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc44006d4b0 con 0x7fc4401457b0 2024-04-24T23:49:26.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.068+0000 7fc448695640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fc44006da40 con 0x7fc44015f990 2024-04-24T23:49:26.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.068+0000 7fc446431640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc440144b20 0x7fc440159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.068+0000 7fc446431640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc440144b20 0x7fc440159f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48766/0 (socket says 172.21.15.16:48766) 2024-04-24T23:49:26.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.068+0000 7fc446431640 1 -- 172.21.15.16:0/23180312 learned_addr learned my addr 172.21.15.16:0/23180312 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:26.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.068+0000 7fc445c30640 1 --2- 172.21.15.16:0/23180312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4401457b0 0x7fc44015e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.068+0000 7fc446c32640 1 --2- 172.21.15.16:0/23180312 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc44015f990 0x7fc4401f1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.069+0000 7fc446431640 1 -- 172.21.15.16:0/23180312 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc44015f990 msgr2=0x7fc4401f1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.069+0000 7fc446431640 1 --2- 172.21.15.16:0/23180312 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc44015f990 0x7fc4401f1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.069+0000 7fc446431640 1 -- 172.21.15.16:0/23180312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4401457b0 msgr2=0x7fc44015e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.069+0000 7fc446431640 1 --2- 172.21.15.16:0/23180312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4401457b0 0x7fc44015e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.069+0000 7fc446431640 1 -- 172.21.15.16:0/23180312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc44015d790 con 0x7fc440144b20 2024-04-24T23:49:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.069+0000 7fc446431640 1 --2- 172.21.15.16:0/23180312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc440144b20 0x7fc440159f20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c400 tx=0x7fc43800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.069+0000 7fc43effd640 1 -- 172.21.15.16:0/23180312 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4380074e0 con 0x7fc440144b20 2024-04-24T23:49:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.069+0000 7fc43effd640 1 -- 172.21.15.16:0/23180312 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc438007680 con 0x7fc440144b20 2024-04-24T23:49:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.069+0000 7fc448695640 1 -- 172.21.15.16:0/23180312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc44015da70 con 0x7fc440144b20 2024-04-24T23:49:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.069+0000 7fc43effd640 1 -- 172.21.15.16:0/23180312 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc438004a00 con 0x7fc440144b20 2024-04-24T23:49:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.069+0000 7fc448695640 1 -- 172.21.15.16:0/23180312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc44015e030 con 0x7fc440144b20 2024-04-24T23:49:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.070+0000 7fc448695640 1 -- 172.21.15.16:0/23180312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4400731f0 con 0x7fc440144b20 2024-04-24T23:49:26.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.071+0000 7fc43effd640 1 -- 172.21.15.16:0/23180312 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc43801a020 con 0x7fc440144b20 2024-04-24T23:49:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.071+0000 7fc43effd640 1 --2- 172.21.15.16:0/23180312 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc42c06be30 0x7fc42c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.071+0000 7fc43effd640 1 -- 172.21.15.16:0/23180312 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fc43808a870 con 0x7fc440144b20 2024-04-24T23:49:26.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.071+0000 7fc445c30640 1 --2- 172.21.15.16:0/23180312 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc42c06be30 0x7fc42c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.072+0000 7fc445c30640 1 --2- 172.21.15.16:0/23180312 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc42c06be30 0x7fc42c06e2f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc434034000 tx=0x7fc43402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:26.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.075+0000 7fc43effd640 1 -- 172.21.15.16:0/23180312 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc438058890 con 0x7fc440144b20 2024-04-24T23:49:26.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.258+0000 7fc448695640 1 -- 172.21.15.16:0/23180312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc44004f5e0 con 0x7fc440144b20 2024-04-24T23:49:26.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.260+0000 7fc43effd640 1 -- 172.21.15.16:0/23180312 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18502 (secure 0 0 0) 0x7fc43805cb90 con 0x7fc440144b20 2024-04-24T23:49:26.259 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:26.260 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:59.888015+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.261+0000 7fc448695640 1 -- 172.21.15.16:0/23180312 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc42c06be30 msgr2=0x7fc42c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.261+0000 7fc448695640 1 --2- 172.21.15.16:0/23180312 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc42c06be30 0x7fc42c06e2f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc434034000 tx=0x7fc43402e000 comp rx=0 tx=0).stop 2024-04-24T23:49:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.261+0000 7fc448695640 1 -- 172.21.15.16:0/23180312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc440144b20 msgr2=0x7fc440159f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.261+0000 7fc448695640 1 --2- 172.21.15.16:0/23180312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc440144b20 0x7fc440159f20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fc43800c400 tx=0x7fc43800c8c0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.262+0000 7fc448695640 1 -- 172.21.15.16:0/23180312 shutdown_connections 2024-04-24T23:49:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.262+0000 7fc448695640 1 --2- 172.21.15.16:0/23180312 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc42c06be30 0x7fc42c06e2f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.262+0000 7fc448695640 1 --2- 172.21.15.16:0/23180312 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc44015f990 0x7fc4401f1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.262+0000 7fc448695640 1 --2- 172.21.15.16:0/23180312 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4401457b0 0x7fc44015e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.262+0000 7fc448695640 1 --2- 172.21.15.16:0/23180312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc440144b20 0x7fc440159f20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.262+0000 7fc448695640 1 -- 172.21.15.16:0/23180312 >> 172.21.15.16:0/23180312 conn(0x7fc4400b2000 msgr2=0x7fc4400cf5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:26.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.262+0000 7fc448695640 1 -- 172.21.15.16:0/23180312 shutdown_connections 2024-04-24T23:49:26.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.262+0000 7fc448695640 1 -- 172.21.15.16:0/23180312 wait complete. 2024-04-24T23:49:26.272 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:26.273 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:26.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.377+0000 7f148b71b640 1 Processor -- start 2024-04-24T23:49:26.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.377+0000 7f148b71b640 1 -- start start 2024-04-24T23:49:26.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.378+0000 7f148b71b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14840d12d0 0x7f14840d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.378+0000 7f148b71b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f148415e5c0 0x7f14840d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.378+0000 7f148b71b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f148415ec30 con 0x7f14840cd790 2024-04-24T23:49:26.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.378+0000 7f148b71b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f148415f4d0 con 0x7f148415e5c0 2024-04-24T23:49:26.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.378+0000 7f148b71b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f148415ef10 con 0x7f14840d12d0 2024-04-24T23:49:26.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.378+0000 7f14894b7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14840d12d0 0x7f14840d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.378+0000 7f1489cb8640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f148415e5c0 0x7f14840d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.378+0000 7f1488cb6640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f14840cd790 0x7f14840c8580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:41384/0 (socket says 172.21.15.16:41384) 2024-04-24T23:49:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.378+0000 7f1488cb6640 1 -- 172.21.15.16:0/2573231553 learned_addr learned my addr 172.21.15.16:0/2573231553 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.378+0000 7f14894b7640 1 -- 172.21.15.16:0/2573231553 >> v2:172.21.15.156:3300/0 conn(0x7f148415e5c0 msgr2=0x7f14840d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.378+0000 7f14894b7640 1 --2- 172.21.15.16:0/2573231553 >> v2:172.21.15.156:3300/0 conn(0x7f148415e5c0 0x7f14840d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.378+0000 7f14894b7640 1 -- 172.21.15.16:0/2573231553 >> v1:172.21.15.156:6789/0 conn(0x7f14840cd790 legacy=0x7f14840c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.378+0000 7f14894b7640 1 -- 172.21.15.16:0/2573231553 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f148415f1f0 con 0x7f14840d12d0 2024-04-24T23:49:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.378+0000 7f1489cb8640 1 --2- 172.21.15.16:0/2573231553 >> v2:172.21.15.156:3300/0 conn(0x7f148415e5c0 0x7f14840d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.379+0000 7f14894b7640 1 --2- 172.21.15.16:0/2573231553 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14840d12d0 0x7f14840d16b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f14800097a0 tx=0x7f148002a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=904464b3b34f55e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.379+0000 7f147bfff640 1 -- 172.21.15.16:0/2573231553 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480037070 con 0x7f14840d12d0 2024-04-24T23:49:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.379+0000 7f147bfff640 1 -- 172.21.15.16:0/2573231553 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f148002adb0 con 0x7f14840d12d0 2024-04-24T23:49:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.379+0000 7f147bfff640 1 -- 172.21.15.16:0/2573231553 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1480032660 con 0x7f14840d12d0 2024-04-24T23:49:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.380+0000 7f148b71b640 1 -- 172.21.15.16:0/2573231553 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14840d12d0 msgr2=0x7f14840d16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.380+0000 7f148b71b640 1 --2- 172.21.15.16:0/2573231553 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14840d12d0 0x7f14840d16b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f14800097a0 tx=0x7f148002a060 comp rx=0 tx=0).stop 2024-04-24T23:49:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.380+0000 7f148b71b640 1 -- 172.21.15.16:0/2573231553 shutdown_connections 2024-04-24T23:49:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.380+0000 7f148b71b640 1 --2- 172.21.15.16:0/2573231553 >> v2:172.21.15.156:3300/0 conn(0x7f148415e5c0 0x7f14840d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.380+0000 7f148b71b640 1 --2- 172.21.15.16:0/2573231553 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14840d12d0 0x7f14840d16b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.380+0000 7f148b71b640 1 -- 172.21.15.16:0/2573231553 >> 172.21.15.16:0/2573231553 conn(0x7f1484072e70 msgr2=0x7f1484073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.380+0000 7f148b71b640 1 -- 172.21.15.16:0/2573231553 shutdown_connections 2024-04-24T23:49:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.380+0000 7f148b71b640 1 -- 172.21.15.16:0/2573231553 wait complete. 2024-04-24T23:49:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.381+0000 7f148b71b640 1 Processor -- start 2024-04-24T23:49:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.381+0000 7f148b71b640 1 -- start start 2024-04-24T23:49:26.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f148b71b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14840cd790 0x7f14841f3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.383 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f148b71b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14840d12d0 0x7f1484157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f148b71b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f148415e5c0 0x7f1484161980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f148b71b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f14840d27f0 con 0x7f148415e5c0 2024-04-24T23:49:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f148b71b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f14840d2670 con 0x7f14840cd790 2024-04-24T23:49:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f148b71b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f14840d2970 con 0x7f14840d12d0 2024-04-24T23:49:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f1489cb8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f148415e5c0 0x7f1484161980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f1489cb8640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f148415e5c0 0x7f1484161980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48774/0 (socket says 172.21.15.16:48774) 2024-04-24T23:49:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f1489cb8640 1 -- 172.21.15.16:0/281186141 learned_addr learned my addr 172.21.15.16:0/281186141 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f1488cb6640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14840d12d0 0x7f1484157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f14894b7640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14840cd790 0x7f14841f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f1489cb8640 1 -- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14840d12d0 msgr2=0x7f1484157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f1489cb8640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14840d12d0 0x7f1484157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f1489cb8640 1 -- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14840cd790 msgr2=0x7f14841f3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f1489cb8640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14840cd790 0x7f14841f3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f1489cb8640 1 -- 172.21.15.16:0/281186141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1484156650 con 0x7f148415e5c0 2024-04-24T23:49:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.382+0000 7f14894b7640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14840cd790 0x7f14841f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.383+0000 7f1489cb8640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f148415e5c0 0x7f1484161980 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f147c00efc0 tx=0x7f147c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.383+0000 7f1479ffb640 1 -- 172.21.15.16:0/281186141 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147c018030 con 0x7f148415e5c0 2024-04-24T23:49:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.383+0000 7f1479ffb640 1 -- 172.21.15.16:0/281186141 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f147c00cdf0 con 0x7f148415e5c0 2024-04-24T23:49:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.383+0000 7f148b71b640 1 -- 172.21.15.16:0/281186141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1484157990 con 0x7f148415e5c0 2024-04-24T23:49:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.383+0000 7f1479ffb640 1 -- 172.21.15.16:0/281186141 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f147c003bb0 con 0x7f148415e5c0 2024-04-24T23:49:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.383+0000 7f148b71b640 1 -- 172.21.15.16:0/281186141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1484156bd0 con 0x7f148415e5c0 2024-04-24T23:49:26.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.384+0000 7f148b71b640 1 -- 172.21.15.16:0/281186141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1484068f90 con 0x7f148415e5c0 2024-04-24T23:49:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.384+0000 7f1479ffb640 1 -- 172.21.15.16:0/281186141 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f147c005d10 con 0x7f148415e5c0 2024-04-24T23:49:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.385+0000 7f1479ffb640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f146806be30 0x7f146806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.385+0000 7f1479ffb640 1 -- 172.21.15.16:0/281186141 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f147c08bb70 con 0x7f148415e5c0 2024-04-24T23:49:26.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.385+0000 7f14894b7640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f146806be30 0x7f146806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.386+0000 7f14894b7640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f146806be30 0x7f146806e2f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1480034000 tx=0x7f148002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.389+0000 7f1479ffb640 1 -- 172.21.15.16:0/281186141 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f147c022070 con 0x7f148415e5c0 2024-04-24T23:49:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.572+0000 7f148b71b640 1 -- 172.21.15.16:0/281186141 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f14840b0b40 con 0x7f148415e5c0 2024-04-24T23:49:26.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.572+0000 7f1479ffb640 1 -- 172.21.15.16:0/281186141 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f147c010040 con 0x7f148415e5c0 2024-04-24T23:49:26.572 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.574+0000 7f148b71b640 1 -- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f146806be30 msgr2=0x7f146806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.574+0000 7f148b71b640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f146806be30 0x7f146806e2f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1480034000 tx=0x7f148002e000 comp rx=0 tx=0).stop 2024-04-24T23:49:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.574+0000 7f148b71b640 1 -- 172.21.15.16:0/281186141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f148415e5c0 msgr2=0x7f1484161980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.574+0000 7f148b71b640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f148415e5c0 0x7f1484161980 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f147c00efc0 tx=0x7f147c00c490 comp rx=0 tx=0).stop 2024-04-24T23:49:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.574+0000 7f148b71b640 1 -- 172.21.15.16:0/281186141 shutdown_connections 2024-04-24T23:49:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.574+0000 7f148b71b640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f146806be30 0x7f146806e2f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.574+0000 7f148b71b640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f148415e5c0 0x7f1484161980 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.574+0000 7f148b71b640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14840d12d0 0x7f1484157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.574+0000 7f148b71b640 1 --2- 172.21.15.16:0/281186141 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14840cd790 0x7f14841f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.574+0000 7f148b71b640 1 -- 172.21.15.16:0/281186141 >> 172.21.15.16:0/281186141 conn(0x7f1484072e70 msgr2=0x7f14840b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.574+0000 7f148b71b640 1 -- 172.21.15.16:0/281186141 shutdown_connections 2024-04-24T23:49:26.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.574+0000 7f148b71b640 1 -- 172.21.15.16:0/281186141 wait complete. 2024-04-24T23:49:26.584 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.688+0000 7ff5963c4640 1 Processor -- start 2024-04-24T23:49:26.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.689+0000 7ff5963c4640 1 -- start start 2024-04-24T23:49:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.689+0000 7ff5963c4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff5900d2260 0x7ff5900b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.689+0000 7ff5963c4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5900b2a10 con 0x7ff5900b21d0 2024-04-24T23:49:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.689+0000 7ff5963c4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5900b2cf0 con 0x7ff5901478d0 2024-04-24T23:49:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.689+0000 7ff5963c4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff5900b3620 con 0x7ff5900d2260 2024-04-24T23:49:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.689+0000 7ff58f7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff5900d2260 0x7ff5900b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.689+0000 7ff594961640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff5900b21d0 0x7ff5900b4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38484/0 (socket says 172.21.15.16:38484) 2024-04-24T23:49:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.689+0000 7ff58f7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff5900d2260 0x7ff5900b1c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49180/0 (socket says 172.21.15.16:49180) 2024-04-24T23:49:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.689+0000 7ff594961640 1 -- 172.21.15.16:0/373583753 learned_addr learned my addr 172.21.15.16:0/373583753 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.690+0000 7ff58f7fe640 1 -- 172.21.15.16:0/373583753 >> v1:172.21.15.156:6789/0 conn(0x7ff5901478d0 legacy=0x7ff5900d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.690+0000 7ff58f7fe640 1 -- 172.21.15.16:0/373583753 >> v1:172.21.15.16:6789/0 conn(0x7ff5900b21d0 legacy=0x7ff5900b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.690+0000 7ff58f7fe640 1 -- 172.21.15.16:0/373583753 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5900b2fd0 con 0x7ff5900d2260 2024-04-24T23:49:26.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.690+0000 7ff58effd640 1 -- 172.21.15.16:0/373583753 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2250857175 0 0) 0x7ff5900b2a10 con 0x7ff5900b21d0 2024-04-24T23:49:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.690+0000 7ff58f7fe640 1 --2- 172.21.15.16:0/373583753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5900d2260 0x7ff5900b1c90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff5840097a0 tx=0x7ff58402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617d50454e4e2f30 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.691+0000 7ff58effd640 1 -- 172.21.15.16:0/373583753 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff584037070 con 0x7ff5900d2260 2024-04-24T23:49:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.691+0000 7ff58effd640 1 -- 172.21.15.16:0/373583753 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff584032440 con 0x7ff5900d2260 2024-04-24T23:49:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.691+0000 7ff58effd640 1 -- 172.21.15.16:0/373583753 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff584032810 con 0x7ff5900d2260 2024-04-24T23:49:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.691+0000 7ff5963c4640 1 -- 172.21.15.16:0/373583753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5900d2260 msgr2=0x7ff5900b1c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.691+0000 7ff5963c4640 1 --2- 172.21.15.16:0/373583753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5900d2260 0x7ff5900b1c90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff5840097a0 tx=0x7ff58402a230 comp rx=0 tx=0).stop 2024-04-24T23:49:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.691+0000 7ff5963c4640 1 -- 172.21.15.16:0/373583753 shutdown_connections 2024-04-24T23:49:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.691+0000 7ff5963c4640 1 --2- 172.21.15.16:0/373583753 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5900d2260 0x7ff5900b1c90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.691+0000 7ff5963c4640 1 -- 172.21.15.16:0/373583753 >> 172.21.15.16:0/373583753 conn(0x7ff59006b2c0 msgr2=0x7ff590072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.691+0000 7ff5963c4640 1 -- 172.21.15.16:0/373583753 shutdown_connections 2024-04-24T23:49:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.691+0000 7ff5963c4640 1 -- 172.21.15.16:0/373583753 wait complete. 2024-04-24T23:49:26.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.692+0000 7ff5963c4640 1 Processor -- start 2024-04-24T23:49:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.693+0000 7ff5963c4640 1 -- start start 2024-04-24T23:49:26.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.693+0000 7ff5963c4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5900b21d0 0x7ff59015d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.693+0000 7ff5963c4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5900d2260 0x7ff5900cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.693+0000 7ff5963c4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5901478d0 0x7ff5900cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.693+0000 7ff5963c4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff5900b27a0 con 0x7ff5900b21d0 2024-04-24T23:49:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.693+0000 7ff5963c4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff5900b2620 con 0x7ff5900d2260 2024-04-24T23:49:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.693+0000 7ff5963c4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff5900b5520 con 0x7ff5901478d0 2024-04-24T23:49:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.693+0000 7ff58ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5900b21d0 0x7ff59015d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.693+0000 7ff58ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5900b21d0 0x7ff59015d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49184/0 (socket says 172.21.15.16:49184) 2024-04-24T23:49:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.693+0000 7ff58ffff640 1 -- 172.21.15.16:0/1063246966 learned_addr learned my addr 172.21.15.16:0/1063246966 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.694+0000 7ff58f7fe640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5900d2260 0x7ff5900cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.694+0000 7ff594961640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5901478d0 0x7ff5900cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.694+0000 7ff58ffff640 1 -- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5901478d0 msgr2=0x7ff5900cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.694+0000 7ff58ffff640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5901478d0 0x7ff5900cd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.694+0000 7ff58ffff640 1 -- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5900d2260 msgr2=0x7ff5900cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.694+0000 7ff58ffff640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5900d2260 0x7ff5900cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.694+0000 7ff58ffff640 1 -- 172.21.15.16:0/1063246966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59015fb80 con 0x7ff5900b21d0 2024-04-24T23:49:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.694+0000 7ff594961640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5901478d0 0x7ff5900cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.694+0000 7ff58ffff640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5900b21d0 0x7ff59015d5c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff58000c400 tx=0x7ff58000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.694+0000 7ff58cff9640 1 -- 172.21.15.16:0/1063246966 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580017070 con 0x7ff5900b21d0 2024-04-24T23:49:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.694+0000 7ff58cff9640 1 -- 172.21.15.16:0/1063246966 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5800052d0 con 0x7ff5900b21d0 2024-04-24T23:49:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.694+0000 7ff5963c4640 1 -- 172.21.15.16:0/1063246966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5900cb0f0 con 0x7ff5900b21d0 2024-04-24T23:49:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.694+0000 7ff58cff9640 1 -- 172.21.15.16:0/1063246966 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580003830 con 0x7ff5900b21d0 2024-04-24T23:49:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.694+0000 7ff5963c4640 1 -- 172.21.15.16:0/1063246966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5900cb6b0 con 0x7ff5900b21d0 2024-04-24T23:49:26.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.695+0000 7ff5963c4640 1 -- 172.21.15.16:0/1063246966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff590068f90 con 0x7ff5900b21d0 2024-04-24T23:49:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.695+0000 7ff58cff9640 1 -- 172.21.15.16:0/1063246966 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff58001a070 con 0x7ff5900b21d0 2024-04-24T23:49:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.696+0000 7ff58cff9640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff57c06be30 0x7ff57c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.696+0000 7ff58cff9640 1 -- 172.21.15.16:0/1063246966 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff58008a480 con 0x7ff5900b21d0 2024-04-24T23:49:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.696+0000 7ff58f7fe640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff57c06be30 0x7ff57c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:26.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.697+0000 7ff58f7fe640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff57c06be30 0x7ff57c06e2f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff584034000 tx=0x7ff58402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:26.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.700+0000 7ff58cff9640 1 -- 172.21.15.16:0/1063246966 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff5800584a0 con 0x7ff5900b21d0 2024-04-24T23:49:26.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.888+0000 7ff5963c4640 1 -- 172.21.15.16:0/1063246966 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff590067250 con 0x7ff5900b21d0 2024-04-24T23:49:26.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.888+0000 7ff58cff9640 1 -- 172.21.15.16:0/1063246966 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7ff58005c7a0 con 0x7ff5900b21d0 2024-04-24T23:49:26.888 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.890+0000 7ff5963c4640 1 -- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff57c06be30 msgr2=0x7ff57c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.890+0000 7ff5963c4640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff57c06be30 0x7ff57c06e2f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff584034000 tx=0x7ff58402e000 comp rx=0 tx=0).stop 2024-04-24T23:49:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.890+0000 7ff5963c4640 1 -- 172.21.15.16:0/1063246966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5900b21d0 msgr2=0x7ff59015d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.890+0000 7ff5963c4640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5900b21d0 0x7ff59015d5c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff58000c400 tx=0x7ff58000c8c0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.890+0000 7ff5963c4640 1 -- 172.21.15.16:0/1063246966 shutdown_connections 2024-04-24T23:49:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.890+0000 7ff5963c4640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff57c06be30 0x7ff57c06e2f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.890+0000 7ff5963c4640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff5901478d0 0x7ff5900cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.890+0000 7ff5963c4640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5900d2260 0x7ff5900cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.890+0000 7ff5963c4640 1 --2- 172.21.15.16:0/1063246966 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff5900b21d0 0x7ff59015d5c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.890+0000 7ff5963c4640 1 -- 172.21.15.16:0/1063246966 >> 172.21.15.16:0/1063246966 conn(0x7ff59006b2c0 msgr2=0x7ff5900b6350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.890+0000 7ff5963c4640 1 -- 172.21.15.16:0/1063246966 shutdown_connections 2024-04-24T23:49:26.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:26.890+0000 7ff5963c4640 1 -- 172.21.15.16:0/1063246966 wait complete. 2024-04-24T23:49:26.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.004+0000 7f73a3a76640 1 Processor -- start 2024-04-24T23:49:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.005+0000 7f73a3a76640 1 -- start start 2024-04-24T23:49:27.005 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.005+0000 7f73a3a76640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f739c1444c0 0x7f739c071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.005+0000 7f73a3a76640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f739c071fd0 0x7f739c0ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.005+0000 7f73a3a76640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f739c0cf0a0 con 0x7f739c0ce870 2024-04-24T23:49:27.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.005+0000 7f73a3a76640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f739c072430 con 0x7f739c1444c0 2024-04-24T23:49:27.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.005+0000 7f73a3a76640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f739c06bbf0 con 0x7f739c071fd0 2024-04-24T23:49:27.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.006+0000 7f73a2013640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f739c0ce870 0x7f739c0b3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38498/0 (socket says 172.21.15.16:38498) 2024-04-24T23:49:27.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.006+0000 7f73a1812640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f739c1444c0 0x7f739c071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.006+0000 7f73a2013640 1 -- 172.21.15.16:0/656902338 learned_addr learned my addr 172.21.15.16:0/656902338 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:27.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.006+0000 7f73a1011640 1 --2- 172.21.15.16:0/656902338 >> v2:172.21.15.156:3300/0 conn(0x7f739c071fd0 0x7f739c0ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.006+0000 7f73a1812640 1 -- 172.21.15.16:0/656902338 >> v2:172.21.15.156:3300/0 conn(0x7f739c071fd0 msgr2=0x7f739c0ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:27.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.006+0000 7f73a1812640 1 --2- 172.21.15.16:0/656902338 >> v2:172.21.15.156:3300/0 conn(0x7f739c071fd0 0x7f739c0ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.006+0000 7f73a1812640 1 -- 172.21.15.16:0/656902338 >> v1:172.21.15.16:6789/0 conn(0x7f739c0ce870 legacy=0x7f739c0b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.006+0000 7f73a1812640 1 -- 172.21.15.16:0/656902338 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739c06be90 con 0x7f739c1444c0 2024-04-24T23:49:27.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.006+0000 7f73a0810640 1 -- 172.21.15.16:0/656902338 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3020009859 0 0) 0x7f739c0cf0a0 con 0x7f739c0ce870 2024-04-24T23:49:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.007+0000 7f73a1812640 1 --2- 172.21.15.16:0/656902338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f739c1444c0 0x7f739c071a90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f739c06ec30 tx=0x7f739802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d36a89e6a18461f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.007+0000 7f73a0810640 1 -- 172.21.15.16:0/656902338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7398037070 con 0x7f739c1444c0 2024-04-24T23:49:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.007+0000 7f73a0810640 1 -- 172.21.15.16:0/656902338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7398032440 con 0x7f739c1444c0 2024-04-24T23:49:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.007+0000 7f73a0810640 1 -- 172.21.15.16:0/656902338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7398032810 con 0x7f739c1444c0 2024-04-24T23:49:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.007+0000 7f73a3a76640 1 -- 172.21.15.16:0/656902338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f739c1444c0 msgr2=0x7f739c071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.007+0000 7f73a3a76640 1 --2- 172.21.15.16:0/656902338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f739c1444c0 0x7f739c071a90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f739c06ec30 tx=0x7f739802a1a0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.007+0000 7f73a3a76640 1 -- 172.21.15.16:0/656902338 shutdown_connections 2024-04-24T23:49:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.007+0000 7f73a3a76640 1 --2- 172.21.15.16:0/656902338 >> v2:172.21.15.156:3300/0 conn(0x7f739c071fd0 0x7f739c0ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.007+0000 7f73a3a76640 1 --2- 172.21.15.16:0/656902338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f739c1444c0 0x7f739c071a90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.007+0000 7f73a3a76640 1 -- 172.21.15.16:0/656902338 >> 172.21.15.16:0/656902338 conn(0x7f739c0cfc50 msgr2=0x7f739c0d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.007+0000 7f73a3a76640 1 -- 172.21.15.16:0/656902338 shutdown_connections 2024-04-24T23:49:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.007+0000 7f73a3a76640 1 -- 172.21.15.16:0/656902338 wait complete. 2024-04-24T23:49:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.008+0000 7f73a3a76640 1 Processor -- start 2024-04-24T23:49:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.009+0000 7f73a3a76640 1 -- start start 2024-04-24T23:49:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.009+0000 7f73a3a76640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f739c071fd0 0x7f739c13a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.009+0000 7f73a3a76640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739c0ce870 0x7f739c1f1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.009+0000 7f73a3a76640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f739c1444c0 0x7f739c1e53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.009+0000 7f73a3a76640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f739c06c730 con 0x7f739c071fd0 2024-04-24T23:49:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.009+0000 7f73a3a76640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f739c06c1d0 con 0x7f739c0ce870 2024-04-24T23:49:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.009+0000 7f73a3a76640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f739c0cecc0 con 0x7f739c1444c0 2024-04-24T23:49:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.009+0000 7f73a1812640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f739c071fd0 0x7f739c13a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.009+0000 7f73a1812640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f739c071fd0 0x7f739c13a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49202/0 (socket says 172.21.15.16:49202) 2024-04-24T23:49:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.009+0000 7f73a1812640 1 -- 172.21.15.16:0/152363750 learned_addr learned my addr 172.21.15.16:0/152363750 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.010+0000 7f73a1812640 1 -- 172.21.15.16:0/152363750 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f739c1444c0 msgr2=0x7f739c1e53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.010+0000 7f73a1812640 1 --2- 172.21.15.16:0/152363750 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f739c1444c0 0x7f739c1e53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.010+0000 7f73a1812640 1 -- 172.21.15.16:0/152363750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739c0ce870 msgr2=0x7f739c1f1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.010+0000 7f73a1812640 1 --2- 172.21.15.16:0/152363750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739c0ce870 0x7f739c1f1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.010+0000 7f73a1812640 1 -- 172.21.15.16:0/152363750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739c142160 con 0x7f739c071fd0 2024-04-24T23:49:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.010+0000 7f73a1812640 1 --2- 172.21.15.16:0/152363750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f739c071fd0 0x7f739c13a310 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7398000c00 tx=0x7f73980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.010+0000 7f73927fc640 1 -- 172.21.15.16:0/152363750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7398037050 con 0x7f739c071fd0 2024-04-24T23:49:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.010+0000 7f73927fc640 1 -- 172.21.15.16:0/152363750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7398038040 con 0x7f739c071fd0 2024-04-24T23:49:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.010+0000 7f73a3a76640 1 -- 172.21.15.16:0/152363750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f739c142440 con 0x7f739c071fd0 2024-04-24T23:49:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.010+0000 7f73927fc640 1 -- 172.21.15.16:0/152363750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7398004470 con 0x7f739c071fd0 2024-04-24T23:49:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.010+0000 7f73a3a76640 1 -- 172.21.15.16:0/152363750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f739c142a00 con 0x7f739c071fd0 2024-04-24T23:49:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.011+0000 7f73a3a76640 1 -- 172.21.15.16:0/152363750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f739c13a8b0 con 0x7f739c071fd0 2024-04-24T23:49:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.011+0000 7f73927fc640 1 -- 172.21.15.16:0/152363750 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7398030900 con 0x7f739c071fd0 2024-04-24T23:49:27.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.012+0000 7f73927fc640 1 --2- 172.21.15.16:0/152363750 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f738006be30 0x7f738006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.012+0000 7f73927fc640 1 -- 172.21.15.16:0/152363750 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f73980abd40 con 0x7f739c071fd0 2024-04-24T23:49:27.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.012+0000 7f73a1011640 1 --2- 172.21.15.16:0/152363750 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f738006be30 0x7f738006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.013+0000 7f73a1011640 1 --2- 172.21.15.16:0/152363750 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f738006be30 0x7f738006e2f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f738c006fd0 tx=0x7f738c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.016+0000 7f73927fc640 1 -- 172.21.15.16:0/152363750 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f739c13a8b0 con 0x7f739c071fd0 2024-04-24T23:49:27.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.198+0000 7f73a3a76640 1 -- 172.21.15.16:0/152363750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f739c068f90 con 0x7f739c071fd0 2024-04-24T23:49:27.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.198+0000 7f73927fc640 1 -- 172.21.15.16:0/152363750 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f739807a560 con 0x7f739c071fd0 2024-04-24T23:49:27.198 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.200+0000 7f73a3a76640 1 -- 172.21.15.16:0/152363750 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f738006be30 msgr2=0x7f738006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.200+0000 7f73a3a76640 1 --2- 172.21.15.16:0/152363750 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f738006be30 0x7f738006e2f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f738c006fd0 tx=0x7f738c008040 comp rx=0 tx=0).stop 2024-04-24T23:49:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.200+0000 7f73a3a76640 1 -- 172.21.15.16:0/152363750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f739c071fd0 msgr2=0x7f739c13a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.200+0000 7f73a3a76640 1 --2- 172.21.15.16:0/152363750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f739c071fd0 0x7f739c13a310 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7398000c00 tx=0x7f73980046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.200+0000 7f73a3a76640 1 -- 172.21.15.16:0/152363750 shutdown_connections 2024-04-24T23:49:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.200+0000 7f73a3a76640 1 --2- 172.21.15.16:0/152363750 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f738006be30 0x7f738006e2f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.200+0000 7f73a3a76640 1 --2- 172.21.15.16:0/152363750 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f739c1444c0 0x7f739c1e53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.200+0000 7f73a3a76640 1 --2- 172.21.15.16:0/152363750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739c0ce870 0x7f739c1f1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.200+0000 7f73a3a76640 1 --2- 172.21.15.16:0/152363750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f739c071fd0 0x7f739c13a310 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.200+0000 7f73a3a76640 1 -- 172.21.15.16:0/152363750 >> 172.21.15.16:0/152363750 conn(0x7f739c0cfc50 msgr2=0x7f739c0b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.200+0000 7f73a3a76640 1 -- 172.21.15.16:0/152363750 shutdown_connections 2024-04-24T23:49:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.200+0000 7f73a3a76640 1 -- 172.21.15.16:0/152363750 wait complete. 2024-04-24T23:49:27.211 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:27.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.314+0000 7ffb1b396640 1 Processor -- start 2024-04-24T23:49:27.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.314+0000 7ffb1b396640 1 -- start start 2024-04-24T23:49:27.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.315+0000 7ffb1b396640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb140d26e0 0x7ffb1406bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.315+0000 7ffb1b396640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ffb140b75c0 0x7ffb140b7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.315+0000 7ffb1b396640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb1415e6d0 con 0x7ffb1415e2f0 2024-04-24T23:49:27.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.315+0000 7ffb1b396640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ffb140d2ca0 con 0x7ffb140b75c0 2024-04-24T23:49:27.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.315+0000 7ffb1b396640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ffb140d2f80 con 0x7ffb140d26e0 2024-04-24T23:49:27.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.315+0000 7ffb18931640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb140d26e0 0x7ffb1406bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.315+0000 7ffb19933640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ffb140b75c0 0x7ffb140b7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.315+0000 7ffb18931640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb140d26e0 0x7ffb1406bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:34426/0 (socket says 172.21.15.16:34426) 2024-04-24T23:49:27.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.315+0000 7ffb18931640 1 -- 172.21.15.16:0/3829185933 learned_addr learned my addr 172.21.15.16:0/3829185933 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:27.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.315+0000 7ffb19933640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ffb140b75c0 0x7ffb140b7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:40432/0 (socket says 172.21.15.16:40432) 2024-04-24T23:49:27.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.315+0000 7ffb19132640 1 --1- 172.21.15.16:0/3829185933 >> v1:172.21.15.156:6789/0 conn(0x7ffb1415e2f0 0x7ffb1406b410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:35192/0 (socket says 172.21.15.16:35192) 2024-04-24T23:49:27.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.316+0000 7ffb18931640 1 -- 172.21.15.16:0/3829185933 >> v2:172.21.15.156:3300/0 conn(0x7ffb140b75c0 msgr2=0x7ffb140b7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:27.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.316+0000 7ffb18931640 1 --2- 172.21.15.16:0/3829185933 >> v2:172.21.15.156:3300/0 conn(0x7ffb140b75c0 0x7ffb140b7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.316+0000 7ffb18931640 1 -- 172.21.15.16:0/3829185933 >> v1:172.21.15.156:6789/0 conn(0x7ffb1415e2f0 legacy=0x7ffb1406b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.316+0000 7ffb18931640 1 -- 172.21.15.16:0/3829185933 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb140d3260 con 0x7ffb140d26e0 2024-04-24T23:49:27.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.316+0000 7ffb18931640 1 --2- 172.21.15.16:0/3829185933 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb140d26e0 0x7ffb1406bb20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009870 tx=0x7ffb0402a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab61aa2a04516b20 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:27.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.317+0000 7ffb0bfff640 1 -- 172.21.15.16:0/3829185933 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb04037070 con 0x7ffb140d26e0 2024-04-24T23:49:27.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.317+0000 7ffb0bfff640 1 -- 172.21.15.16:0/3829185933 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb0402ae50 con 0x7ffb140d26e0 2024-04-24T23:49:27.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.317+0000 7ffb0bfff640 1 -- 172.21.15.16:0/3829185933 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb04032860 con 0x7ffb140d26e0 2024-04-24T23:49:27.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.317+0000 7ffb1b396640 1 -- 172.21.15.16:0/3829185933 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb140d26e0 msgr2=0x7ffb1406bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.317+0000 7ffb1b396640 1 --2- 172.21.15.16:0/3829185933 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb140d26e0 0x7ffb1406bb20 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffb04009870 tx=0x7ffb0402a300 comp rx=0 tx=0).stop 2024-04-24T23:49:27.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.317+0000 7ffb1b396640 1 -- 172.21.15.16:0/3829185933 shutdown_connections 2024-04-24T23:49:27.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.317+0000 7ffb1b396640 1 --2- 172.21.15.16:0/3829185933 >> v2:172.21.15.156:3300/0 conn(0x7ffb140b75c0 0x7ffb140b7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.317+0000 7ffb1b396640 1 --2- 172.21.15.16:0/3829185933 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb140d26e0 0x7ffb1406bb20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.317+0000 7ffb1b396640 1 -- 172.21.15.16:0/3829185933 >> 172.21.15.16:0/3829185933 conn(0x7ffb140d4850 msgr2=0x7ffb140d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:27.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.317+0000 7ffb1b396640 1 -- 172.21.15.16:0/3829185933 shutdown_connections 2024-04-24T23:49:27.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.317+0000 7ffb1b396640 1 -- 172.21.15.16:0/3829185933 wait complete. 2024-04-24T23:49:27.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.318+0000 7ffb1b396640 1 Processor -- start 2024-04-24T23:49:27.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.318+0000 7ffb1b396640 1 -- start start 2024-04-24T23:49:27.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.319+0000 7ffb1b396640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb140b75c0 0x7ffb140cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.319+0000 7ffb1b396640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb140d26e0 0x7ffb1415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.319+0000 7ffb1b396640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb1415e2f0 0x7ffb140b2ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.319+0000 7ffb1b396640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffb140d0080 con 0x7ffb140b75c0 2024-04-24T23:49:27.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.319+0000 7ffb1b396640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ffb140b70a0 con 0x7ffb140d26e0 2024-04-24T23:49:27.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.319+0000 7ffb1b396640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ffb1406c060 con 0x7ffb1415e2f0 2024-04-24T23:49:27.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.319+0000 7ffb19132640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb140b75c0 0x7ffb140cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.319+0000 7ffb19132640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb140b75c0 0x7ffb140cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49206/0 (socket says 172.21.15.16:49206) 2024-04-24T23:49:27.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.319+0000 7ffb19132640 1 -- 172.21.15.16:0/2920398793 learned_addr learned my addr 172.21.15.16:0/2920398793 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:27.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.319+0000 7ffb18931640 1 --2- 172.21.15.16:0/2920398793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb140d26e0 0x7ffb1415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.319+0000 7ffb19933640 1 --2- 172.21.15.16:0/2920398793 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb1415e2f0 0x7ffb140b2ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.320+0000 7ffb19132640 1 -- 172.21.15.16:0/2920398793 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb1415e2f0 msgr2=0x7ffb140b2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.320+0000 7ffb19132640 1 --2- 172.21.15.16:0/2920398793 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb1415e2f0 0x7ffb140b2ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.320+0000 7ffb19132640 1 -- 172.21.15.16:0/2920398793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb140d26e0 msgr2=0x7ffb1415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.320+0000 7ffb19132640 1 --2- 172.21.15.16:0/2920398793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb140d26e0 0x7ffb1415a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.320+0000 7ffb19132640 1 -- 172.21.15.16:0/2920398793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb140cb3f0 con 0x7ffb140b75c0 2024-04-24T23:49:27.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.320+0000 7ffb19132640 1 --2- 172.21.15.16:0/2920398793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb140b75c0 0x7ffb140cae10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ffb1000c3c0 tx=0x7ffb1000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:27.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.320+0000 7ffb09ffb640 1 -- 172.21.15.16:0/2920398793 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb100074e0 con 0x7ffb140b75c0 2024-04-24T23:49:27.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.320+0000 7ffb1b396640 1 -- 172.21.15.16:0/2920398793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb140cb6d0 con 0x7ffb140b75c0 2024-04-24T23:49:27.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.320+0000 7ffb09ffb640 1 -- 172.21.15.16:0/2920398793 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb10007680 con 0x7ffb140b75c0 2024-04-24T23:49:27.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.320+0000 7ffb09ffb640 1 -- 172.21.15.16:0/2920398793 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb10004a00 con 0x7ffb140b75c0 2024-04-24T23:49:27.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.320+0000 7ffb1b396640 1 -- 172.21.15.16:0/2920398793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb140cbc90 con 0x7ffb140b75c0 2024-04-24T23:49:27.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.321+0000 7ffb1b396640 1 -- 172.21.15.16:0/2920398793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb14068f90 con 0x7ffb140b75c0 2024-04-24T23:49:27.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.322+0000 7ffb09ffb640 1 -- 172.21.15.16:0/2920398793 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ffb1001a020 con 0x7ffb140b75c0 2024-04-24T23:49:27.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.322+0000 7ffb09ffb640 1 --2- 172.21.15.16:0/2920398793 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffaf806be30 0x7ffaf806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.322+0000 7ffb09ffb640 1 -- 172.21.15.16:0/2920398793 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ffb1008a850 con 0x7ffb140b75c0 2024-04-24T23:49:27.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.322+0000 7ffb18931640 1 --2- 172.21.15.16:0/2920398793 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffaf806be30 0x7ffaf806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.323 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.323+0000 7ffb18931640 1 --2- 172.21.15.16:0/2920398793 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffaf806be30 0x7ffaf806e2f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffb04034000 tx=0x7ffb0402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:27.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.326+0000 7ffb09ffb640 1 -- 172.21.15.16:0/2920398793 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ffb100587f0 con 0x7ffb140b75c0 2024-04-24T23:49:27.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.510+0000 7ffb1b396640 1 -- 172.21.15.16:0/2920398793 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ffb140d2220 con 0x7ffb140b75c0 2024-04-24T23:49:27.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.510+0000 7ffb09ffb640 1 -- 172.21.15.16:0/2920398793 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7ffb1005caf0 con 0x7ffb140b75c0 2024-04-24T23:49:27.510 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:27.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.512+0000 7ffb1b396640 1 -- 172.21.15.16:0/2920398793 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffaf806be30 msgr2=0x7ffaf806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.512+0000 7ffb1b396640 1 --2- 172.21.15.16:0/2920398793 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffaf806be30 0x7ffaf806e2f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ffb04034000 tx=0x7ffb0402e000 comp rx=0 tx=0).stop 2024-04-24T23:49:27.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.512+0000 7ffb1b396640 1 -- 172.21.15.16:0/2920398793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb140b75c0 msgr2=0x7ffb140cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.512+0000 7ffb1b396640 1 --2- 172.21.15.16:0/2920398793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb140b75c0 0x7ffb140cae10 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ffb1000c3c0 tx=0x7ffb1000c880 comp rx=0 tx=0).stop 2024-04-24T23:49:27.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.512+0000 7ffb1b396640 1 -- 172.21.15.16:0/2920398793 shutdown_connections 2024-04-24T23:49:27.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.512+0000 7ffb1b396640 1 --2- 172.21.15.16:0/2920398793 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffaf806be30 0x7ffaf806e2f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.512+0000 7ffb1b396640 1 --2- 172.21.15.16:0/2920398793 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffb1415e2f0 0x7ffb140b2ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.512+0000 7ffb1b396640 1 --2- 172.21.15.16:0/2920398793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffb140d26e0 0x7ffb1415a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.512+0000 7ffb1b396640 1 --2- 172.21.15.16:0/2920398793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb140b75c0 0x7ffb140cae10 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.512+0000 7ffb1b396640 1 -- 172.21.15.16:0/2920398793 >> 172.21.15.16:0/2920398793 conn(0x7ffb140d4850 msgr2=0x7ffb140b5400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:27.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.512+0000 7ffb1b396640 1 -- 172.21.15.16:0/2920398793 shutdown_connections 2024-04-24T23:49:27.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.512+0000 7ffb1b396640 1 -- 172.21.15.16:0/2920398793 wait complete. 2024-04-24T23:49:27.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:49:27.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.624+0000 7fdfbc99e640 1 Processor -- start 2024-04-24T23:49:27.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.624+0000 7fdfbc99e640 1 -- start start 2024-04-24T23:49:27.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfbc99e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfb806cb00 0x7fdfb815c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfbc99e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdfb80ccc70 0x7fdfb8072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfbc99e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfb80cfd50 con 0x7fdfb80727d0 2024-04-24T23:49:27.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfbc99e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fdfb8072c40 con 0x7fdfb806cb00 2024-04-24T23:49:27.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfbc99e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fdfb80ce760 con 0x7fdfb80ccc70 2024-04-24T23:49:27.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfb659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfb806cb00 0x7fdfb815c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfb6d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdfb80727d0 0x7fdfb815f8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38500/0 (socket says 172.21.15.16:38500) 2024-04-24T23:49:27.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfb6d9d640 1 -- 172.21.15.16:0/805634888 learned_addr learned my addr 172.21.15.16:0/805634888 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:27.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfaffff640 1 --2- 172.21.15.16:0/805634888 >> v2:172.21.15.156:3300/0 conn(0x7fdfb80ccc70 0x7fdfb8072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfb659c640 1 -- 172.21.15.16:0/805634888 >> v2:172.21.15.156:3300/0 conn(0x7fdfb80ccc70 msgr2=0x7fdfb8072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:27.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfb659c640 1 --2- 172.21.15.16:0/805634888 >> v2:172.21.15.156:3300/0 conn(0x7fdfb80ccc70 0x7fdfb8072290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfb659c640 1 -- 172.21.15.16:0/805634888 >> v1:172.21.15.16:6789/0 conn(0x7fdfb80727d0 legacy=0x7fdfb815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfb5d9b640 1 -- 172.21.15.16:0/805634888 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3157945706 0 0) 0x7fdfb80cfd50 con 0x7fdfb80727d0 2024-04-24T23:49:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfb659c640 1 -- 172.21.15.16:0/805634888 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb806cee0 con 0x7fdfb806cb00 2024-04-24T23:49:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.625+0000 7fdfaffff640 1 --2- 172.21.15.16:0/805634888 >> v2:172.21.15.156:3300/0 conn(0x7fdfb80ccc70 0x7fdfb8072290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.626+0000 7fdfb659c640 1 --2- 172.21.15.16:0/805634888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb806cb00 0x7fdfb815c250 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8074620 tx=0x7fdfa802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d928bab29d52fffc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.626+0000 7fdfb5d9b640 1 -- 172.21.15.16:0/805634888 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8037070 con 0x7fdfb806cb00 2024-04-24T23:49:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.626+0000 7fdfb5d9b640 1 -- 172.21.15.16:0/805634888 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfa8032440 con 0x7fdfb806cb00 2024-04-24T23:49:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.626+0000 7fdfb5d9b640 1 -- 172.21.15.16:0/805634888 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8032810 con 0x7fdfb806cb00 2024-04-24T23:49:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.627+0000 7fdfbc99e640 1 -- 172.21.15.16:0/805634888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb806cb00 msgr2=0x7fdfb815c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.627+0000 7fdfbc99e640 1 --2- 172.21.15.16:0/805634888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb806cb00 0x7fdfb815c250 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdfb8074620 tx=0x7fdfa802a1a0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.627+0000 7fdfbc99e640 1 -- 172.21.15.16:0/805634888 shutdown_connections 2024-04-24T23:49:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.627+0000 7fdfbc99e640 1 --2- 172.21.15.16:0/805634888 >> v2:172.21.15.156:3300/0 conn(0x7fdfb80ccc70 0x7fdfb8072290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.627+0000 7fdfbc99e640 1 --2- 172.21.15.16:0/805634888 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb806cb00 0x7fdfb815c250 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.627+0000 7fdfbc99e640 1 -- 172.21.15.16:0/805634888 >> 172.21.15.16:0/805634888 conn(0x7fdfb80d0260 msgr2=0x7fdfb80d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.627+0000 7fdfbc99e640 1 -- 172.21.15.16:0/805634888 shutdown_connections 2024-04-24T23:49:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.627+0000 7fdfbc99e640 1 -- 172.21.15.16:0/805634888 wait complete. 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.628+0000 7fdfbc99e640 1 Processor -- start 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.628+0000 7fdfbc99e640 1 -- start start 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfbc99e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfb806cb00 0x7fdfb8142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfbc99e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb80727d0 0x7fdfb8159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfbc99e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfb80ccc70 0x7fdfb815acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfbc99e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb815ce70 con 0x7fdfb80727d0 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfbc99e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb815cb90 con 0x7fdfb80ccc70 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfbc99e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fdfb815d150 con 0x7fdfb806cb00 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfaffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb80727d0 0x7fdfb8159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfaffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb80727d0 0x7fdfb8159310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49212/0 (socket says 172.21.15.16:49212) 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfaffff640 1 -- 172.21.15.16:0/2831138523 learned_addr learned my addr 172.21.15.16:0/2831138523 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfb659c640 1 --2- 172.21.15.16:0/2831138523 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfb806cb00 0x7fdfb8142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfb6d9d640 1 --2- 172.21.15.16:0/2831138523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfb80ccc70 0x7fdfb815acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfaffff640 1 -- 172.21.15.16:0/2831138523 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfb806cb00 msgr2=0x7fdfb8142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfaffff640 1 --2- 172.21.15.16:0/2831138523 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfb806cb00 0x7fdfb8142cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfaffff640 1 -- 172.21.15.16:0/2831138523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfb80ccc70 msgr2=0x7fdfb815acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfaffff640 1 --2- 172.21.15.16:0/2831138523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfb80ccc70 0x7fdfb815acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfaffff640 1 -- 172.21.15.16:0/2831138523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb81431f0 con 0x7fdfb80727d0 2024-04-24T23:49:27.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfaffff640 1 --2- 172.21.15.16:0/2831138523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb80727d0 0x7fdfb8159310 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000c7f0 tx=0x7fdfa000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:27.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfaeffd640 1 -- 172.21.15.16:0/2831138523 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0015070 con 0x7fdfb80727d0 2024-04-24T23:49:27.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.629+0000 7fdfaeffd640 1 -- 172.21.15.16:0/2831138523 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfa0007a30 con 0x7fdfb80727d0 2024-04-24T23:49:27.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.630+0000 7fdfbc99e640 1 -- 172.21.15.16:0/2831138523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb8159c10 con 0x7fdfb80727d0 2024-04-24T23:49:27.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.630+0000 7fdfaeffd640 1 -- 172.21.15.16:0/2831138523 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa0007e00 con 0x7fdfb80727d0 2024-04-24T23:49:27.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.630+0000 7fdfbc99e640 1 -- 172.21.15.16:0/2831138523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb815a1d0 con 0x7fdfb80727d0 2024-04-24T23:49:27.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.631+0000 7fdfbc99e640 1 -- 172.21.15.16:0/2831138523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb804f5e0 con 0x7fdfb80727d0 2024-04-24T23:49:27.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.631+0000 7fdfaeffd640 1 -- 172.21.15.16:0/2831138523 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdfa0018070 con 0x7fdfb80727d0 2024-04-24T23:49:27.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.631+0000 7fdfaeffd640 1 --2- 172.21.15.16:0/2831138523 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdf9806be60 0x7fdf9806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.632+0000 7fdfaeffd640 1 -- 172.21.15.16:0/2831138523 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fdfa008b1c0 con 0x7fdfb80727d0 2024-04-24T23:49:27.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.632+0000 7fdfb659c640 1 --2- 172.21.15.16:0/2831138523 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdf9806be60 0x7fdf9806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.632+0000 7fdfb659c640 1 --2- 172.21.15.16:0/2831138523 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdf9806be60 0x7fdf9806e320 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8034000 tx=0x7fdfa802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:27.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.636+0000 7fdfaeffd640 1 -- 172.21.15.16:0/2831138523 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdfb81431f0 con 0x7fdfb80727d0 2024-04-24T23:49:27.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.855+0000 7fdfbc99e640 1 -- 172.21.15.16:0/2831138523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdfb8072e90 con 0x7fdfb80727d0 2024-04-24T23:49:27.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.856+0000 7fdfaeffd640 1 -- 172.21.15.16:0/2831138523 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5180 (secure 0 0 0) 0x7fdfa0059790 con 0x7fdfb80727d0 2024-04-24T23:49:27.855 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:27.856 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1623032341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1623032341},{"type":"v1","addr":"172.21.15.156:6836","nonce":1623032341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3530738236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3530738236},{"type":"v1","addr":"172.21.15.16:6835","nonce":3530738236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2153063131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2153063131},{"type":"v1","addr":"172.21.15.16:6834","nonce":2153063131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:48:54.852787+0000","modified":"2024-04-24T23:49:17.914730+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/289127714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":289127714},{"type":"v1","addr":"172.21.15.156:6837","nonce":289127714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-24T23:49:27.856 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-04-24T23:49:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.857+0000 7fdfbc99e640 1 -- 172.21.15.16:0/2831138523 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdf9806be60 msgr2=0x7fdf9806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.857+0000 7fdfbc99e640 1 --2- 172.21.15.16:0/2831138523 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdf9806be60 0x7fdf9806e320 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fdfa8034000 tx=0x7fdfa802e000 comp rx=0 tx=0).stop 2024-04-24T23:49:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.857+0000 7fdfbc99e640 1 -- 172.21.15.16:0/2831138523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb80727d0 msgr2=0x7fdfb8159310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.857+0000 7fdfbc99e640 1 --2- 172.21.15.16:0/2831138523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb80727d0 0x7fdfb8159310 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fdfa000c7f0 tx=0x7fdfa000cbf0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.858+0000 7fdfbc99e640 1 -- 172.21.15.16:0/2831138523 shutdown_connections 2024-04-24T23:49:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.858+0000 7fdfbc99e640 1 --2- 172.21.15.16:0/2831138523 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdf9806be60 0x7fdf9806e320 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.858+0000 7fdfbc99e640 1 --2- 172.21.15.16:0/2831138523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdfb80ccc70 0x7fdfb815acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.858+0000 7fdfbc99e640 1 --2- 172.21.15.16:0/2831138523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfb80727d0 0x7fdfb8159310 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.858+0000 7fdfbc99e640 1 --2- 172.21.15.16:0/2831138523 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdfb806cb00 0x7fdfb8142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.858+0000 7fdfbc99e640 1 -- 172.21.15.16:0/2831138523 >> 172.21.15.16:0/2831138523 conn(0x7fdfb80d0260 msgr2=0x7fdfb8071280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.858+0000 7fdfbc99e640 1 -- 172.21.15.16:0/2831138523 shutdown_connections 2024-04-24T23:49:27.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.858+0000 7fdfbc99e640 1 -- 172.21.15.16:0/2831138523 wait complete. 2024-04-24T23:49:27.869 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:27.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.969+0000 7ff0a7f63640 1 Processor -- start 2024-04-24T23:49:27.969 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.970+0000 7ff0a7f63640 1 -- start start 2024-04-24T23:49:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a7f63640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0a015df20 0x7ff0a0071a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a7f63640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff0a00d0530 0x7ff0a0071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a7f63640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff0a00cfe50 0x7ff0a00ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a7f63640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff0a00cf460 con 0x7ff0a00cfe50 2024-04-24T23:49:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a7f63640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ff0a00b6820 con 0x7ff0a00d0530 2024-04-24T23:49:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a7f63640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff0a00b6a70 con 0x7ff0a015df20 2024-04-24T23:49:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a6500640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff0a00cfe50 0x7ff0a00ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a6500640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff0a00cfe50 0x7ff0a00ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49228/0 (socket says 172.21.15.16:49228) 2024-04-24T23:49:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a6500640 1 -- 172.21.15.16:0/1997095340 learned_addr learned my addr 172.21.15.16:0/1997095340 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a5cff640 1 --2- 172.21.15.16:0/1997095340 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0a015df20 0x7ff0a0071a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a54fe640 1 --2- 172.21.15.16:0/1997095340 >> v2:172.21.15.156:3300/0 conn(0x7ff0a00d0530 0x7ff0a0071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a6500640 1 -- 172.21.15.16:0/1997095340 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0a015df20 msgr2=0x7ff0a0071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a6500640 1 --2- 172.21.15.16:0/1997095340 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0a015df20 0x7ff0a0071a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a6500640 1 -- 172.21.15.16:0/1997095340 >> v2:172.21.15.156:3300/0 conn(0x7ff0a00d0530 msgr2=0x7ff0a0071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a6500640 1 --2- 172.21.15.16:0/1997095340 >> v2:172.21.15.156:3300/0 conn(0x7ff0a00d0530 0x7ff0a0071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a6500640 1 -- 172.21.15.16:0/1997095340 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a00ce6a0 con 0x7ff0a00cfe50 2024-04-24T23:49:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.971+0000 7ff0a54fe640 1 --2- 172.21.15.16:0/1997095340 >> v2:172.21.15.156:3300/0 conn(0x7ff0a00d0530 0x7ff0a0071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.972+0000 7ff0a6500640 1 --2- 172.21.15.16:0/1997095340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a00cfe50 0x7ff0a00ad8e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0068be0 tx=0x7ff09002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8478dbad235b5a6f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.972+0000 7ff0a4cfd640 1 -- 172.21.15.16:0/1997095340 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff090036070 con 0x7ff0a00cfe50 2024-04-24T23:49:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.972+0000 7ff0a4cfd640 1 -- 172.21.15.16:0/1997095340 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff090031600 con 0x7ff0a00cfe50 2024-04-24T23:49:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.972+0000 7ff0a4cfd640 1 -- 172.21.15.16:0/1997095340 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0900319d0 con 0x7ff0a00cfe50 2024-04-24T23:49:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.972+0000 7ff0a7f63640 1 -- 172.21.15.16:0/1997095340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a00cfe50 msgr2=0x7ff0a00ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.972+0000 7ff0a7f63640 1 --2- 172.21.15.16:0/1997095340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a00cfe50 0x7ff0a00ad8e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff0a0068be0 tx=0x7ff09002a6a0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.973+0000 7ff0a7f63640 1 -- 172.21.15.16:0/1997095340 shutdown_connections 2024-04-24T23:49:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.973+0000 7ff0a7f63640 1 --2- 172.21.15.16:0/1997095340 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a00cfe50 0x7ff0a00ad8e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.973+0000 7ff0a7f63640 1 --2- 172.21.15.16:0/1997095340 >> v2:172.21.15.156:3300/0 conn(0x7ff0a00d0530 0x7ff0a0071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.973+0000 7ff0a7f63640 1 --2- 172.21.15.16:0/1997095340 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0a015df20 0x7ff0a0071a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.973+0000 7ff0a7f63640 1 -- 172.21.15.16:0/1997095340 >> 172.21.15.16:0/1997095340 conn(0x7ff0a00d0ac0 msgr2=0x7ff0a00d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.973+0000 7ff0a7f63640 1 -- 172.21.15.16:0/1997095340 shutdown_connections 2024-04-24T23:49:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.973+0000 7ff0a7f63640 1 -- 172.21.15.16:0/1997095340 wait complete. 2024-04-24T23:49:27.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.974+0000 7ff0a7f63640 1 Processor -- start 2024-04-24T23:49:27.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.974+0000 7ff0a7f63640 1 -- start start 2024-04-24T23:49:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a7f63640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0a00cfe50 0x7ff0a015f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a7f63640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a00d0530 0x7ff0a015fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a7f63640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0a015df20 0x7ff0a00b5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a7f63640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a00cef40 con 0x7ff0a00d0530 2024-04-24T23:49:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a7f63640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff0a00cec60 con 0x7ff0a00cfe50 2024-04-24T23:49:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a7f63640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff0a00b4030 con 0x7ff0a015df20 2024-04-24T23:49:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a54fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a00d0530 0x7ff0a015fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a54fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a00d0530 0x7ff0a015fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49236/0 (socket says 172.21.15.16:49236) 2024-04-24T23:49:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a54fe640 1 -- 172.21.15.16:0/2748944865 learned_addr learned my addr 172.21.15.16:0/2748944865 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a6500640 1 --2- 172.21.15.16:0/2748944865 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0a015df20 0x7ff0a00b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a5cff640 1 --2- 172.21.15.16:0/2748944865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0a00cfe50 0x7ff0a015f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a54fe640 1 -- 172.21.15.16:0/2748944865 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0a015df20 msgr2=0x7ff0a00b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a54fe640 1 --2- 172.21.15.16:0/2748944865 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0a015df20 0x7ff0a00b5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a54fe640 1 -- 172.21.15.16:0/2748944865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0a00cfe50 msgr2=0x7ff0a015f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a54fe640 1 --2- 172.21.15.16:0/2748944865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0a00cfe50 0x7ff0a015f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a54fe640 1 -- 172.21.15.16:0/2748944865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0a006c960 con 0x7ff0a00d0530 2024-04-24T23:49:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a54fe640 1 --2- 172.21.15.16:0/2748944865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a00d0530 0x7ff0a015fd50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff09400e7e0 tx=0x7ff09400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff09e7fc640 1 -- 172.21.15.16:0/2748944865 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff094017070 con 0x7ff0a00d0530 2024-04-24T23:49:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff09e7fc640 1 -- 172.21.15.16:0/2748944865 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff09400c6d0 con 0x7ff0a00d0530 2024-04-24T23:49:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a7f63640 1 -- 172.21.15.16:0/2748944865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0a015e3a0 con 0x7ff0a00d0530 2024-04-24T23:49:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff09e7fc640 1 -- 172.21.15.16:0/2748944865 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff09400f040 con 0x7ff0a00d0530 2024-04-24T23:49:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.975+0000 7ff0a7f63640 1 -- 172.21.15.16:0/2748944865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0a015e960 con 0x7ff0a00d0530 2024-04-24T23:49:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.976+0000 7ff0a7f63640 1 -- 172.21.15.16:0/2748944865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0a006f8f0 con 0x7ff0a00d0530 2024-04-24T23:49:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.977+0000 7ff09e7fc640 1 -- 172.21.15.16:0/2748944865 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff09401a070 con 0x7ff0a00d0530 2024-04-24T23:49:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.977+0000 7ff09e7fc640 1 --2- 172.21.15.16:0/2748944865 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff08c06be30 0x7ff08c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.977+0000 7ff09e7fc640 1 -- 172.21.15.16:0/2748944865 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff09408b060 con 0x7ff0a00d0530 2024-04-24T23:49:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.977+0000 7ff0a5cff640 1 --2- 172.21.15.16:0/2748944865 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff08c06be30 0x7ff08c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.978+0000 7ff0a5cff640 1 --2- 172.21.15.16:0/2748944865 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff08c06be30 0x7ff08c06e2f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff098006fd0 tx=0x7ff098008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:27.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:27.981+0000 7ff09e7fc640 1 -- 172.21.15.16:0/2748944865 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff094059080 con 0x7ff0a00d0530 2024-04-24T23:49:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.168+0000 7ff0a7f63640 1 -- 172.21.15.16:0/2748944865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff0a0069500 con 0x7ff0a00d0530 2024-04-24T23:49:28.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.169+0000 7ff09e7fc640 1 -- 172.21.15.16:0/2748944865 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18502 (secure 0 0 0) 0x7ff09405d380 con 0x7ff0a00d0530 2024-04-24T23:49:28.169 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:28.169 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:59.888015+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.171+0000 7ff0a7f63640 1 -- 172.21.15.16:0/2748944865 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff08c06be30 msgr2=0x7ff08c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.171+0000 7ff0a7f63640 1 --2- 172.21.15.16:0/2748944865 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff08c06be30 0x7ff08c06e2f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7ff098006fd0 tx=0x7ff098008040 comp rx=0 tx=0).stop 2024-04-24T23:49:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.171+0000 7ff0a7f63640 1 -- 172.21.15.16:0/2748944865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a00d0530 msgr2=0x7ff0a015fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.171+0000 7ff0a7f63640 1 --2- 172.21.15.16:0/2748944865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a00d0530 0x7ff0a015fd50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff09400e7e0 tx=0x7ff09400eca0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.171+0000 7ff0a7f63640 1 -- 172.21.15.16:0/2748944865 shutdown_connections 2024-04-24T23:49:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.171+0000 7ff0a7f63640 1 --2- 172.21.15.16:0/2748944865 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff08c06be30 0x7ff08c06e2f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.171+0000 7ff0a7f63640 1 --2- 172.21.15.16:0/2748944865 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff0a015df20 0x7ff0a00b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.171+0000 7ff0a7f63640 1 --2- 172.21.15.16:0/2748944865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0a00d0530 0x7ff0a015fd50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.171+0000 7ff0a7f63640 1 --2- 172.21.15.16:0/2748944865 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff0a00cfe50 0x7ff0a015f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.171+0000 7ff0a7f63640 1 -- 172.21.15.16:0/2748944865 >> 172.21.15.16:0/2748944865 conn(0x7ff0a00d0ac0 msgr2=0x7ff0a00b3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.171+0000 7ff0a7f63640 1 -- 172.21.15.16:0/2748944865 shutdown_connections 2024-04-24T23:49:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.171+0000 7ff0a7f63640 1 -- 172.21.15.16:0/2748944865 wait complete. 2024-04-24T23:49:28.182 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:28.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:28.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.287+0000 7f0302148640 1 Processor -- start 2024-04-24T23:49:28.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.287+0000 7f0302148640 1 -- start start 2024-04-24T23:49:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.288+0000 7f0302148640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02fc0cd790 0x7f02fc0d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.288+0000 7f0302148640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02fc15e5c0 0x7f02fc0d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.288+0000 7f0302148640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02fc15ec30 con 0x7f02fc0d12d0 2024-04-24T23:49:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.288+0000 7f0302148640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f02fc15f4d0 con 0x7f02fc15e5c0 2024-04-24T23:49:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.288+0000 7f0302148640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f02fc15ef10 con 0x7f02fc0cd790 2024-04-24T23:49:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.288+0000 7f02fbfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02fc15e5c0 0x7f02fc0d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.288+0000 7f02fbfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02fc15e5c0 0x7f02fc0d2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49240/0 (socket says 172.21.15.16:49240) 2024-04-24T23:49:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.288+0000 7f02fbfff640 1 -- 172.21.15.16:0/367215327 learned_addr learned my addr 172.21.15.16:0/367215327 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.288+0000 7f02faffd640 1 --2- 172.21.15.16:0/367215327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02fc0cd790 0x7f02fc0d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.288+0000 7f02fbfff640 1 -- 172.21.15.16:0/367215327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02fc0cd790 msgr2=0x7f02fc0d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:28.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.288+0000 7f02fbfff640 1 --2- 172.21.15.16:0/367215327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02fc0cd790 0x7f02fc0d16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.288+0000 7f02fbfff640 1 -- 172.21.15.16:0/367215327 >> v1:172.21.15.156:6789/0 conn(0x7f02fc0d12d0 legacy=0x7f02fc0c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.288+0000 7f02fbfff640 1 -- 172.21.15.16:0/367215327 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02fc15f1f0 con 0x7f02fc15e5c0 2024-04-24T23:49:28.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.289+0000 7f02fbfff640 1 --2- 172.21.15.16:0/367215327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc15e5c0 0x7f02fc0d2130 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009e50 tx=0x7f02f402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d6306f5eb62a8e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:28.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.289+0000 7f02fa7fc640 1 -- 172.21.15.16:0/367215327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4036070 con 0x7f02fc15e5c0 2024-04-24T23:49:28.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.289+0000 7f02fa7fc640 1 -- 172.21.15.16:0/367215327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02f4031600 con 0x7f02fc15e5c0 2024-04-24T23:49:28.290 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.289+0000 7f02fa7fc640 1 -- 172.21.15.16:0/367215327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f40319d0 con 0x7f02fc15e5c0 2024-04-24T23:49:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.289+0000 7f0302148640 1 -- 172.21.15.16:0/367215327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc15e5c0 msgr2=0x7f02fc0d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.289+0000 7f0302148640 1 --2- 172.21.15.16:0/367215327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc15e5c0 0x7f02fc0d2130 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009e50 tx=0x7f02f402a5f0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.290+0000 7f0302148640 1 -- 172.21.15.16:0/367215327 shutdown_connections 2024-04-24T23:49:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.290+0000 7f0302148640 1 --2- 172.21.15.16:0/367215327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02fc0cd790 0x7f02fc0d16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.290+0000 7f0302148640 1 --2- 172.21.15.16:0/367215327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc15e5c0 0x7f02fc0d2130 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.290+0000 7f0302148640 1 -- 172.21.15.16:0/367215327 >> 172.21.15.16:0/367215327 conn(0x7f02fc072e70 msgr2=0x7f02fc073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.290+0000 7f0302148640 1 -- 172.21.15.16:0/367215327 shutdown_connections 2024-04-24T23:49:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.290+0000 7f0302148640 1 -- 172.21.15.16:0/367215327 wait complete. 2024-04-24T23:49:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.291+0000 7f0302148640 1 Processor -- start 2024-04-24T23:49:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.291+0000 7f0302148640 1 -- start start 2024-04-24T23:49:28.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f0302148640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02fc0cd790 0x7f02fc1518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f0302148640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02fc0d12d0 0x7f02fc0cb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f0302148640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc15e5c0 0x7f02fc0af550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f0302148640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc0d27f0 con 0x7f02fc15e5c0 2024-04-24T23:49:28.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f0302148640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc0d2670 con 0x7f02fc0d12d0 2024-04-24T23:49:28.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f0302148640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f02fc0d2970 con 0x7f02fc0cd790 2024-04-24T23:49:28.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f02fbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc15e5c0 0x7f02fc0af550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f02fbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc15e5c0 0x7f02fc0af550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49250/0 (socket says 172.21.15.16:49250) 2024-04-24T23:49:28.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f02fbfff640 1 -- 172.21.15.16:0/4238110490 learned_addr learned my addr 172.21.15.16:0/4238110490 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:28.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f02faffd640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02fc0d12d0 0x7f02fc0cb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f02fbfff640 1 -- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02fc0cd790 msgr2=0x7f02fc1518d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f02fb7fe640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02fc0cd790 0x7f02fc1518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f02fbfff640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02fc0cd790 0x7f02fc1518d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f02fbfff640 1 -- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02fc0d12d0 msgr2=0x7f02fc0cb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f02fbfff640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02fc0d12d0 0x7f02fc0cb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f02fbfff640 1 -- 172.21.15.16:0/4238110490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02fc0ad8e0 con 0x7f02fc15e5c0 2024-04-24T23:49:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f02faffd640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02fc0d12d0 0x7f02fc0cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.292+0000 7f02fbfff640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc15e5c0 0x7f02fc0af550 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009f70 tx=0x7f02f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.293+0000 7f02e3fff640 1 -- 172.21.15.16:0/4238110490 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f403f020 con 0x7f02fc15e5c0 2024-04-24T23:49:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.293+0000 7f02e3fff640 1 -- 172.21.15.16:0/4238110490 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02f4037040 con 0x7f02fc15e5c0 2024-04-24T23:49:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.293+0000 7f0302148640 1 -- 172.21.15.16:0/4238110490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02fc0adb80 con 0x7f02fc15e5c0 2024-04-24T23:49:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.293+0000 7f02e3fff640 1 -- 172.21.15.16:0/4238110490 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02f4036070 con 0x7f02fc15e5c0 2024-04-24T23:49:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.293+0000 7f0302148640 1 -- 172.21.15.16:0/4238110490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02fc0ae140 con 0x7f02fc15e5c0 2024-04-24T23:49:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.294+0000 7f0302148640 1 -- 172.21.15.16:0/4238110490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02fc068f90 con 0x7f02fc15e5c0 2024-04-24T23:49:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.294+0000 7f02e3fff640 1 -- 172.21.15.16:0/4238110490 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f02f4042020 con 0x7f02fc15e5c0 2024-04-24T23:49:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.294+0000 7f02e3fff640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02e806be30 0x7f02e806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.295+0000 7f02e3fff640 1 -- 172.21.15.16:0/4238110490 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f02f40ab6a0 con 0x7f02fc15e5c0 2024-04-24T23:49:28.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.295+0000 7f02fb7fe640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02e806be30 0x7f02e806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.295+0000 7f02fb7fe640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02e806be30 0x7f02e806e2f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0b7010 tx=0x7f02ec009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:28.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.299+0000 7f02e3fff640 1 -- 172.21.15.16:0/4238110490 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f02f4049030 con 0x7f02fc15e5c0 2024-04-24T23:49:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.483+0000 7f0302148640 1 -- 172.21.15.16:0/4238110490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f02fc15ff50 con 0x7f02fc15e5c0 2024-04-24T23:49:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.483+0000 7f02e3fff640 1 -- 172.21.15.16:0/4238110490 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) ==== 93+0+10 (secure 0 0 0) 0x7f02f4079d10 con 0x7f02fc15e5c0 2024-04-24T23:49:28.483 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.486+0000 7f0302148640 1 -- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02e806be30 msgr2=0x7f02e806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.486+0000 7f0302148640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02e806be30 0x7f02e806e2f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f02fc0b7010 tx=0x7f02ec009040 comp rx=0 tx=0).stop 2024-04-24T23:49:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.486+0000 7f0302148640 1 -- 172.21.15.16:0/4238110490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc15e5c0 msgr2=0x7f02fc0af550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.486+0000 7f0302148640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc15e5c0 0x7f02fc0af550 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009f70 tx=0x7f02f4004a40 comp rx=0 tx=0).stop 2024-04-24T23:49:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.486+0000 7f0302148640 1 -- 172.21.15.16:0/4238110490 shutdown_connections 2024-04-24T23:49:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.486+0000 7f0302148640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02e806be30 0x7f02e806e2f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.486+0000 7f0302148640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02fc15e5c0 0x7f02fc0af550 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.486+0000 7f0302148640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02fc0d12d0 0x7f02fc0cb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.486+0000 7f0302148640 1 --2- 172.21.15.16:0/4238110490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02fc0cd790 0x7f02fc1518d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.486+0000 7f0302148640 1 -- 172.21.15.16:0/4238110490 >> 172.21.15.16:0/4238110490 conn(0x7f02fc072e70 msgr2=0x7f02fc161100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.486+0000 7f0302148640 1 -- 172.21.15.16:0/4238110490 shutdown_connections 2024-04-24T23:49:28.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.486+0000 7f0302148640 1 -- 172.21.15.16:0/4238110490 wait complete. 2024-04-24T23:49:28.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.600+0000 7f763a9ee640 1 Processor -- start 2024-04-24T23:49:28.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.600+0000 7f763a9ee640 1 -- start start 2024-04-24T23:49:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.601+0000 7f763a9ee640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7634144620 0x7f76340d0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.601+0000 7f763a9ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76340b2aa0 0x7f76340d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.601+0000 7f763a9ee640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76340d0870 con 0x7f76340d2030 2024-04-24T23:49:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.601+0000 7f763a9ee640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f76340d2480 con 0x7f7634144620 2024-04-24T23:49:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.601+0000 7f763a9ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f76340b1a00 con 0x7f76340b2aa0 2024-04-24T23:49:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.601+0000 7f7633fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7634144620 0x7f76340d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.601+0000 7f7633fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7634144620 0x7f76340d0330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49260/0 (socket says 172.21.15.16:49260) 2024-04-24T23:49:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.601+0000 7f7633fff640 1 -- 172.21.15.16:0/3869666242 learned_addr learned my addr 172.21.15.16:0/3869666242 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.602+0000 7f76337fe640 1 --2- 172.21.15.16:0/3869666242 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76340b2aa0 0x7f76340d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.602+0000 7f7633fff640 1 -- 172.21.15.16:0/3869666242 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76340b2aa0 msgr2=0x7f76340d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.602+0000 7f7633fff640 1 --2- 172.21.15.16:0/3869666242 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76340b2aa0 0x7f76340d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.602+0000 7f7633fff640 1 -- 172.21.15.16:0/3869666242 >> v1:172.21.15.156:6789/0 conn(0x7f76340d2030 legacy=0x7f763406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.602+0000 7f7633fff640 1 -- 172.21.15.16:0/3869666242 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76340b1ce0 con 0x7f7634144620 2024-04-24T23:49:28.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.602+0000 7f7633fff640 1 --2- 172.21.15.16:0/3869666242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7634144620 0x7f76340d0330 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7624009870 tx=0x7f762402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561618ef312722a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.603+0000 7f7632ffd640 1 -- 172.21.15.16:0/3869666242 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7624037070 con 0x7f7634144620 2024-04-24T23:49:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.603+0000 7f7632ffd640 1 -- 172.21.15.16:0/3869666242 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7624032440 con 0x7f7634144620 2024-04-24T23:49:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.603+0000 7f7632ffd640 1 -- 172.21.15.16:0/3869666242 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7624032810 con 0x7f7634144620 2024-04-24T23:49:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.603+0000 7f763a9ee640 1 -- 172.21.15.16:0/3869666242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7634144620 msgr2=0x7f76340d0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.603+0000 7f763a9ee640 1 --2- 172.21.15.16:0/3869666242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7634144620 0x7f76340d0330 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7624009870 tx=0x7f762402a270 comp rx=0 tx=0).stop 2024-04-24T23:49:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.603+0000 7f763a9ee640 1 -- 172.21.15.16:0/3869666242 shutdown_connections 2024-04-24T23:49:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.603+0000 7f763a9ee640 1 --2- 172.21.15.16:0/3869666242 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f76340b2aa0 0x7f76340d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.603+0000 7f763a9ee640 1 --2- 172.21.15.16:0/3869666242 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7634144620 0x7f76340d0330 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.603+0000 7f763a9ee640 1 -- 172.21.15.16:0/3869666242 >> 172.21.15.16:0/3869666242 conn(0x7f763406b0e0 msgr2=0x7f763406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.603+0000 7f763a9ee640 1 -- 172.21.15.16:0/3869666242 shutdown_connections 2024-04-24T23:49:28.604 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.603+0000 7f763a9ee640 1 -- 172.21.15.16:0/3869666242 wait complete. 2024-04-24T23:49:28.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.604+0000 7f763a9ee640 1 Processor -- start 2024-04-24T23:49:28.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.605+0000 7f763a9ee640 1 -- start start 2024-04-24T23:49:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.605+0000 7f763a9ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76340b2aa0 0x7f76340b72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.605+0000 7f763a9ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76340d2030 0x7f763413ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.605+0000 7f763a9ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7634144620 0x7f76340b6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.605+0000 7f763a9ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f76340b2580 con 0x7f76340d2030 2024-04-24T23:49:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.605+0000 7f763a9ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f76340b22a0 con 0x7f76340b2aa0 2024-04-24T23:49:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.605+0000 7f763a9ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7634144a00 con 0x7f7634144620 2024-04-24T23:49:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.605+0000 7f76337fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76340d2030 0x7f763413ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.605+0000 7f76337fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76340d2030 0x7f763413ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49272/0 (socket says 172.21.15.16:49272) 2024-04-24T23:49:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.605+0000 7f76337fe640 1 -- 172.21.15.16:0/610779063 learned_addr learned my addr 172.21.15.16:0/610779063 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.606+0000 7f7633fff640 1 --2- 172.21.15.16:0/610779063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76340b2aa0 0x7f76340b72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.606+0000 7f7638f8b640 1 --2- 172.21.15.16:0/610779063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7634144620 0x7f76340b6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.606+0000 7f76337fe640 1 -- 172.21.15.16:0/610779063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7634144620 msgr2=0x7f76340b6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.606+0000 7f76337fe640 1 --2- 172.21.15.16:0/610779063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7634144620 0x7f76340b6250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.606+0000 7f76337fe640 1 -- 172.21.15.16:0/610779063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76340b2aa0 msgr2=0x7f76340b72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.606+0000 7f76337fe640 1 --2- 172.21.15.16:0/610779063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76340b2aa0 0x7f76340b72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.606+0000 7f76337fe640 1 -- 172.21.15.16:0/610779063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76341e8070 con 0x7f76340d2030 2024-04-24T23:49:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.606+0000 7f76337fe640 1 --2- 172.21.15.16:0/610779063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76340d2030 0x7f763413ca10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f762800e9e0 tx=0x7f762800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.606+0000 7f7630ff9640 1 -- 172.21.15.16:0/610779063 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7628017070 con 0x7f76340d2030 2024-04-24T23:49:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.606+0000 7f7630ff9640 1 -- 172.21.15.16:0/610779063 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f762800c970 con 0x7f76340d2030 2024-04-24T23:49:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.606+0000 7f763a9ee640 1 -- 172.21.15.16:0/610779063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76340b5490 con 0x7f76340d2030 2024-04-24T23:49:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.606+0000 7f7630ff9640 1 -- 172.21.15.16:0/610779063 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f762800cd60 con 0x7f76340d2030 2024-04-24T23:49:28.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.606+0000 7f763a9ee640 1 -- 172.21.15.16:0/610779063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76340b5a50 con 0x7f76340d2030 2024-04-24T23:49:28.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.607+0000 7f763a9ee640 1 -- 172.21.15.16:0/610779063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7634068f90 con 0x7f76340d2030 2024-04-24T23:49:28.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.608+0000 7f7630ff9640 1 -- 172.21.15.16:0/610779063 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f762801a070 con 0x7f76340d2030 2024-04-24T23:49:28.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.608+0000 7f7630ff9640 1 --2- 172.21.15.16:0/610779063 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f762006be30 0x7f762006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.608+0000 7f7630ff9640 1 -- 172.21.15.16:0/610779063 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f762808b090 con 0x7f76340d2030 2024-04-24T23:49:28.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.608+0000 7f7633fff640 1 --2- 172.21.15.16:0/610779063 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f762006be30 0x7f762006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.609+0000 7f7633fff640 1 --2- 172.21.15.16:0/610779063 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f762006be30 0x7f762006e2f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7624034000 tx=0x7f762402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:28.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.612+0000 7f7630ff9640 1 -- 172.21.15.16:0/610779063 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f76341e8070 con 0x7f76340d2030 2024-04-24T23:49:28.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.794+0000 7f763a9ee640 1 -- 172.21.15.16:0/610779063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f76340cbb80 con 0x7f76340d2030 2024-04-24T23:49:28.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.794+0000 7f7630ff9640 1 -- 172.21.15.16:0/610779063 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) ==== 104+0+11 (secure 0 0 0) 0x7f76280590b0 con 0x7f76340d2030 2024-04-24T23:49:28.794 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:28.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.796+0000 7f763a9ee640 1 -- 172.21.15.16:0/610779063 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f762006be30 msgr2=0x7f762006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.796+0000 7f763a9ee640 1 --2- 172.21.15.16:0/610779063 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f762006be30 0x7f762006e2f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7624034000 tx=0x7f762402e000 comp rx=0 tx=0).stop 2024-04-24T23:49:28.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.796+0000 7f763a9ee640 1 -- 172.21.15.16:0/610779063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76340d2030 msgr2=0x7f763413ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.796+0000 7f763a9ee640 1 --2- 172.21.15.16:0/610779063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76340d2030 0x7f763413ca10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f762800e9e0 tx=0x7f762800eea0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.796+0000 7f763a9ee640 1 -- 172.21.15.16:0/610779063 shutdown_connections 2024-04-24T23:49:28.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.796+0000 7f763a9ee640 1 --2- 172.21.15.16:0/610779063 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f762006be30 0x7f762006e2f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.796+0000 7f763a9ee640 1 --2- 172.21.15.16:0/610779063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7634144620 0x7f76340b6250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.796+0000 7f763a9ee640 1 --2- 172.21.15.16:0/610779063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76340d2030 0x7f763413ca10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.796+0000 7f763a9ee640 1 --2- 172.21.15.16:0/610779063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f76340b2aa0 0x7f76340b72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.796+0000 7f763a9ee640 1 -- 172.21.15.16:0/610779063 >> 172.21.15.16:0/610779063 conn(0x7f763406b0e0 msgr2=0x7f763406ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:28.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.797+0000 7f763a9ee640 1 -- 172.21.15.16:0/610779063 shutdown_connections 2024-04-24T23:49:28.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.797+0000 7f763a9ee640 1 -- 172.21.15.16:0/610779063 wait complete. 2024-04-24T23:49:28.807 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.912+0000 7f140ad09640 1 Processor -- start 2024-04-24T23:49:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.912+0000 7f140ad09640 1 -- start start 2024-04-24T23:49:28.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.913+0000 7f140ad09640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1404144620 0x7f14040d0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.913+0000 7f140ad09640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14040d0870 con 0x7f14040d2200 2024-04-24T23:49:28.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.913+0000 7f140ad09640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14040b1a00 con 0x7f14040b2aa0 2024-04-24T23:49:28.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.913+0000 7f140ad09640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f14040b1ce0 con 0x7f1404144620 2024-04-24T23:49:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.913+0000 7f14092a6640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f14040d2200 0x7f140406ca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38502/0 (socket says 172.21.15.16:38502) 2024-04-24T23:49:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.913+0000 7f14092a6640 1 -- 172.21.15.16:0/2286685967 learned_addr learned my addr 172.21.15.16:0/2286685967 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 -- 172.21.15.16:0/2286685967 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3918606657 0 0) 0x7f14040d0870 con 0x7f14040d2200 2024-04-24T23:49:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f1408aa5640 1 --2- 172.21.15.16:0/2286685967 >> v2:172.21.15.156:3300/0 conn(0x7f1404144620 0x7f14040d0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 -- 172.21.15.16:0/2286685967 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13ec003520 con 0x7f14040d2200 2024-04-24T23:49:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 -- 172.21.15.16:0/2286685967 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2548863213 0 0) 0x7f13ec003520 con 0x7f14040d2200 2024-04-24T23:49:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 -- 172.21.15.16:0/2286685967 >> v2:172.21.15.156:3300/0 conn(0x7f1404144620 msgr2=0x7f14040d0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 --2- 172.21.15.16:0/2286685967 >> v2:172.21.15.156:3300/0 conn(0x7f1404144620 0x7f14040d0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 -- 172.21.15.16:0/2286685967 >> v1:172.21.15.156:6789/0 conn(0x7f14040b2aa0 legacy=0x7f14040d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 -- 172.21.15.16:0/2286685967 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14040b1fc0 con 0x7f14040d2200 2024-04-24T23:49:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 -- 172.21.15.16:0/2286685967 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f13f4001670 con 0x7f14040d2200 2024-04-24T23:49:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 -- 172.21.15.16:0/2286685967 >> v1:172.21.15.16:6789/0 conn(0x7f14040d2200 legacy=0x7f140406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 --2- 172.21.15.16:0/2286685967 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f13ec0039b0 0x7f13ec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 --2- 172.21.15.16:0/2286685967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13ec004800 0x7f13ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 --2- 172.21.15.16:0/2286685967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec0051c0 0x7f13ec00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 -- 172.21.15.16:0/2286685967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f14040b1ce0 con 0x7f13ec0051c0 2024-04-24T23:49:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 -- 172.21.15.16:0/2286685967 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f13ec00c060 con 0x7f13ec004800 2024-04-24T23:49:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14037fe640 1 -- 172.21.15.16:0/2286685967 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f13ec00c260 con 0x7f13ec0039b0 2024-04-24T23:49:28.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14092a6640 1 --2- 172.21.15.16:0/2286685967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec0051c0 0x7f13ec00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f1403fff640 1 --2- 172.21.15.16:0/2286685967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13ec004800 0x7f13ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f1408aa5640 1 --2- 172.21.15.16:0/2286685967 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f13ec0039b0 0x7f13ec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14092a6640 1 -- 172.21.15.16:0/2286685967 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f13ec0039b0 msgr2=0x7f13ec003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14092a6640 1 --2- 172.21.15.16:0/2286685967 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f13ec0039b0 0x7f13ec003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14092a6640 1 -- 172.21.15.16:0/2286685967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13ec004800 msgr2=0x7f13ec004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14092a6640 1 --2- 172.21.15.16:0/2286685967 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13ec004800 0x7f13ec004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f14092a6640 1 -- 172.21.15.16:0/2286685967 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13ec003520 con 0x7f13ec0051c0 2024-04-24T23:49:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.914+0000 7f1408aa5640 1 --2- 172.21.15.16:0/2286685967 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f13ec0039b0 0x7f13ec003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.915+0000 7f14092a6640 1 -- 172.21.15.16:0/2286685967 reap_dead start 2024-04-24T23:49:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.915+0000 7f14092a6640 1 --2- 172.21.15.16:0/2286685967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec0051c0 0x7f13ec00b9c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f13f400c240 tx=0x7f13ec005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532be613016d9667 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.915+0000 7f14037fe640 1 -- 172.21.15.16:0/2286685967 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4007cf0 con 0x7f13ec0051c0 2024-04-24T23:49:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.915+0000 7f14037fe640 1 -- 172.21.15.16:0/2286685967 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13f40058d0 con 0x7f13ec0051c0 2024-04-24T23:49:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.915+0000 7f14037fe640 1 -- 172.21.15.16:0/2286685967 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4005cc0 con 0x7f13ec0051c0 2024-04-24T23:49:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.916+0000 7f140ad09640 1 -- 172.21.15.16:0/2286685967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec0051c0 msgr2=0x7f13ec00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.916+0000 7f140ad09640 1 --2- 172.21.15.16:0/2286685967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec0051c0 0x7f13ec00b9c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f13f400c240 tx=0x7f13ec005830 comp rx=0 tx=0).stop 2024-04-24T23:49:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.916+0000 7f140ad09640 1 -- 172.21.15.16:0/2286685967 shutdown_connections 2024-04-24T23:49:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.916+0000 7f140ad09640 1 --2- 172.21.15.16:0/2286685967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13ec0051c0 0x7f13ec00b9c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.916+0000 7f140ad09640 1 -- 172.21.15.16:0/2286685967 >> 172.21.15.16:0/2286685967 conn(0x7f140406b0e0 msgr2=0x7f140406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.916+0000 7f140ad09640 1 -- 172.21.15.16:0/2286685967 shutdown_connections 2024-04-24T23:49:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.916+0000 7f140ad09640 1 -- 172.21.15.16:0/2286685967 wait complete. 2024-04-24T23:49:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.917+0000 7f140ad09640 1 Processor -- start 2024-04-24T23:49:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.917+0000 7f140ad09640 1 -- start start 2024-04-24T23:49:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.918+0000 7f140ad09640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13ec0051c0 0x7f1404161650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.918+0000 7f140ad09640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1404160230 0x7f140415dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.918+0000 7f140ad09640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14040cdc20 0x7f14040ce0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.918+0000 7f140ad09640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1404144a00 con 0x7f1404160230 2024-04-24T23:49:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.918+0000 7f140ad09640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f14040b2580 con 0x7f13ec0051c0 2024-04-24T23:49:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.918+0000 7f140ad09640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f14040b1660 con 0x7f14040cdc20 2024-04-24T23:49:28.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.918+0000 7f1403fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1404160230 0x7f140415dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.918+0000 7f1403fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1404160230 0x7f140415dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49280/0 (socket says 172.21.15.16:49280) 2024-04-24T23:49:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.918+0000 7f1403fff640 1 -- 172.21.15.16:0/2738947054 learned_addr learned my addr 172.21.15.16:0/2738947054 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.918+0000 7f14092a6640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14040cdc20 0x7f14040ce0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.918+0000 7f1408aa5640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13ec0051c0 0x7f1404161650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.919+0000 7f1403fff640 1 -- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14040cdc20 msgr2=0x7f14040ce0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.919+0000 7f1403fff640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14040cdc20 0x7f14040ce0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.919+0000 7f1403fff640 1 -- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13ec0051c0 msgr2=0x7f1404161650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.919+0000 7f1403fff640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13ec0051c0 0x7f1404161650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.919+0000 7f1403fff640 1 -- 172.21.15.16:0/2738947054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1404160cf0 con 0x7f1404160230 2024-04-24T23:49:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.919+0000 7f1408aa5640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13ec0051c0 0x7f1404161650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.919+0000 7f14092a6640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14040cdc20 0x7f14040ce0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.919+0000 7f1403fff640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1404160230 0x7f140415dd70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f13f800f5f0 tx=0x7f13f800fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:28.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.919+0000 7f14017fa640 1 -- 172.21.15.16:0/2738947054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f80052e0 con 0x7f1404160230 2024-04-24T23:49:28.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.919+0000 7f14017fa640 1 -- 172.21.15.16:0/2738947054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13f8018070 con 0x7f1404160230 2024-04-24T23:49:28.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.919+0000 7f140ad09640 1 -- 172.21.15.16:0/2738947054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f140415e2b0 con 0x7f1404160230 2024-04-24T23:49:28.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.919+0000 7f14017fa640 1 -- 172.21.15.16:0/2738947054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f8003810 con 0x7f1404160230 2024-04-24T23:49:28.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.919+0000 7f140ad09640 1 -- 172.21.15.16:0/2738947054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1404137a80 con 0x7f1404160230 2024-04-24T23:49:28.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.920+0000 7f140ad09640 1 -- 172.21.15.16:0/2738947054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1404137870 con 0x7f1404160230 2024-04-24T23:49:28.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.920+0000 7f14017fa640 1 -- 172.21.15.16:0/2738947054 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f13f80039b0 con 0x7f1404160230 2024-04-24T23:49:28.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.921+0000 7f14017fa640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f13f006be30 0x7f13f006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:28.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.921+0000 7f14017fa640 1 -- 172.21.15.16:0/2738947054 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f13f808af90 con 0x7f1404160230 2024-04-24T23:49:28.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.921+0000 7f1408aa5640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f13f006be30 0x7f13f006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:28.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.922+0000 7f1408aa5640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f13f006be30 0x7f13f006e2f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f13fc007380 tx=0x7f13fc00b040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:28.933 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:28.925+0000 7f14017fa640 1 -- 172.21.15.16:0/2738947054 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f13f80595f0 con 0x7f1404160230 2024-04-24T23:49:29.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.107+0000 7f140ad09640 1 -- 172.21.15.16:0/2738947054 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1404068f90 con 0x7f1404160230 2024-04-24T23:49:29.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.107+0000 7f14017fa640 1 -- 172.21.15.16:0/2738947054 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) ==== 100+0+11 (secure 0 0 0) 0x7f13f8058ef0 con 0x7f1404160230 2024-04-24T23:49:29.107 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:29.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.109+0000 7f140ad09640 1 -- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f13f006be30 msgr2=0x7f13f006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.109+0000 7f140ad09640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f13f006be30 0x7f13f006e2f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f13fc007380 tx=0x7f13fc00b040 comp rx=0 tx=0).stop 2024-04-24T23:49:29.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.109+0000 7f140ad09640 1 -- 172.21.15.16:0/2738947054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1404160230 msgr2=0x7f140415dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.109+0000 7f140ad09640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1404160230 0x7f140415dd70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f13f800f5f0 tx=0x7f13f800fab0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.109+0000 7f140ad09640 1 -- 172.21.15.16:0/2738947054 shutdown_connections 2024-04-24T23:49:29.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.109+0000 7f140ad09640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f13f006be30 0x7f13f006e2f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.109+0000 7f140ad09640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14040cdc20 0x7f14040ce0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.109+0000 7f140ad09640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1404160230 0x7f140415dd70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.109+0000 7f140ad09640 1 --2- 172.21.15.16:0/2738947054 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f13ec0051c0 0x7f1404161650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.109+0000 7f140ad09640 1 -- 172.21.15.16:0/2738947054 >> 172.21.15.16:0/2738947054 conn(0x7f140406b0e0 msgr2=0x7f140406b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:29.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.109+0000 7f140ad09640 1 -- 172.21.15.16:0/2738947054 shutdown_connections 2024-04-24T23:49:29.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.109+0000 7f140ad09640 1 -- 172.21.15.16:0/2738947054 wait complete. 2024-04-24T23:49:29.120 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:29.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.224+0000 7ff47fd4f640 1 Processor -- start 2024-04-24T23:49:29.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.224+0000 7ff47fd4f640 1 -- start start 2024-04-24T23:49:29.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.225+0000 7ff47fd4f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4781478d0 0x7ff4780d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.225+0000 7ff47fd4f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4780d2090 0x7ff4780b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.225+0000 7ff47fd4f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4780b2a10 con 0x7ff4780b21d0 2024-04-24T23:49:29.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.225+0000 7ff47fd4f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff4780d24f0 con 0x7ff4781478d0 2024-04-24T23:49:29.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.225+0000 7ff47fd4f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff4780b3620 con 0x7ff4780d2090 2024-04-24T23:49:29.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.225+0000 7ff47daeb640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4781478d0 0x7ff4780d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.225+0000 7ff47daeb640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff4781478d0 0x7ff4780d1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49284/0 (socket says 172.21.15.16:49284) 2024-04-24T23:49:29.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.225+0000 7ff47daeb640 1 -- 172.21.15.16:0/794135689 learned_addr learned my addr 172.21.15.16:0/794135689 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:29.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.225+0000 7ff47d2ea640 1 --2- 172.21.15.16:0/794135689 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4780d2090 0x7ff4780b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.225+0000 7ff47daeb640 1 -- 172.21.15.16:0/794135689 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4780d2090 msgr2=0x7ff4780b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:29.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.225+0000 7ff47daeb640 1 --2- 172.21.15.16:0/794135689 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4780d2090 0x7ff4780b1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.225+0000 7ff47daeb640 1 -- 172.21.15.16:0/794135689 >> v1:172.21.15.156:6789/0 conn(0x7ff4780b21d0 legacy=0x7ff4780d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:29.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.225+0000 7ff47daeb640 1 -- 172.21.15.16:0/794135689 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4780b2cf0 con 0x7ff4781478d0 2024-04-24T23:49:29.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.225+0000 7ff47d2ea640 1 --2- 172.21.15.16:0/794135689 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4780d2090 0x7ff4780b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:29.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.226+0000 7ff47daeb640 1 --2- 172.21.15.16:0/794135689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4781478d0 0x7ff4780d1b50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff474009870 tx=0x7ff47402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193823ac047baf3d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:29.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.226+0000 7ff47cae9640 1 -- 172.21.15.16:0/794135689 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff474037070 con 0x7ff4781478d0 2024-04-24T23:49:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.226+0000 7ff47cae9640 1 -- 172.21.15.16:0/794135689 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff474032440 con 0x7ff4781478d0 2024-04-24T23:49:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.226+0000 7ff47cae9640 1 -- 172.21.15.16:0/794135689 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff474032810 con 0x7ff4781478d0 2024-04-24T23:49:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.227+0000 7ff47fd4f640 1 -- 172.21.15.16:0/794135689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4781478d0 msgr2=0x7ff4780d1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.227+0000 7ff47fd4f640 1 --2- 172.21.15.16:0/794135689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4781478d0 0x7ff4780d1b50 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff474009870 tx=0x7ff47402a210 comp rx=0 tx=0).stop 2024-04-24T23:49:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.227+0000 7ff47fd4f640 1 -- 172.21.15.16:0/794135689 shutdown_connections 2024-04-24T23:49:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.227+0000 7ff47fd4f640 1 --2- 172.21.15.16:0/794135689 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4780d2090 0x7ff4780b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.227+0000 7ff47fd4f640 1 --2- 172.21.15.16:0/794135689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4781478d0 0x7ff4780d1b50 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.227+0000 7ff47fd4f640 1 -- 172.21.15.16:0/794135689 >> 172.21.15.16:0/794135689 conn(0x7ff47806b2c0 msgr2=0x7ff478072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.227+0000 7ff47fd4f640 1 -- 172.21.15.16:0/794135689 shutdown_connections 2024-04-24T23:49:29.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.227+0000 7ff47fd4f640 1 -- 172.21.15.16:0/794135689 wait complete. 2024-04-24T23:49:29.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.228+0000 7ff47fd4f640 1 Processor -- start 2024-04-24T23:49:29.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.228+0000 7ff47fd4f640 1 -- start start 2024-04-24T23:49:29.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47fd4f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4780b21d0 0x7ff478140410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47fd4f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4780d2090 0x7ff478140950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47fd4f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4781478d0 0x7ff47815e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47fd4f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff4780b2660 con 0x7ff4780b21d0 2024-04-24T23:49:29.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47fd4f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff4780b32b0 con 0x7ff4781478d0 2024-04-24T23:49:29.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47fd4f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff47813ae20 con 0x7ff4780d2090 2024-04-24T23:49:29.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47daeb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4780b21d0 0x7ff478140410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47daeb640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4780b21d0 0x7ff478140410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49292/0 (socket says 172.21.15.16:49292) 2024-04-24T23:49:29.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47daeb640 1 -- 172.21.15.16:0/3001783539 learned_addr learned my addr 172.21.15.16:0/3001783539 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:29.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47e2ec640 1 --2- 172.21.15.16:0/3001783539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4781478d0 0x7ff47815e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47d2ea640 1 --2- 172.21.15.16:0/3001783539 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4780d2090 0x7ff478140950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47daeb640 1 -- 172.21.15.16:0/3001783539 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4780d2090 msgr2=0x7ff478140950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47daeb640 1 --2- 172.21.15.16:0/3001783539 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4780d2090 0x7ff478140950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47daeb640 1 -- 172.21.15.16:0/3001783539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4781478d0 msgr2=0x7ff47815e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47daeb640 1 --2- 172.21.15.16:0/3001783539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4781478d0 0x7ff47815e9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47daeb640 1 -- 172.21.15.16:0/3001783539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff478141170 con 0x7ff4780b21d0 2024-04-24T23:49:29.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.229+0000 7ff47daeb640 1 --2- 172.21.15.16:0/3001783539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4780b21d0 0x7ff478140410 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff474009870 tx=0x7ff4740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:29.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.230+0000 7ff46e7fc640 1 -- 172.21.15.16:0/3001783539 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff474041070 con 0x7ff4780b21d0 2024-04-24T23:49:29.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.230+0000 7ff46e7fc640 1 -- 172.21.15.16:0/3001783539 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff47403c870 con 0x7ff4780b21d0 2024-04-24T23:49:29.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.230+0000 7ff47fd4f640 1 -- 172.21.15.16:0/3001783539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff478141450 con 0x7ff4780b21d0 2024-04-24T23:49:29.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.230+0000 7ff46e7fc640 1 -- 172.21.15.16:0/3001783539 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff474037040 con 0x7ff4780b21d0 2024-04-24T23:49:29.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.230+0000 7ff47fd4f640 1 -- 172.21.15.16:0/3001783539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff478141a10 con 0x7ff4780b21d0 2024-04-24T23:49:29.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.230+0000 7ff47fd4f640 1 -- 172.21.15.16:0/3001783539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff478068f90 con 0x7ff4780b21d0 2024-04-24T23:49:29.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.231+0000 7ff46e7fc640 1 -- 172.21.15.16:0/3001783539 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff474044070 con 0x7ff4780b21d0 2024-04-24T23:49:29.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.232+0000 7ff46e7fc640 1 --2- 172.21.15.16:0/3001783539 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff45c06be30 0x7ff45c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.232+0000 7ff46e7fc640 1 -- 172.21.15.16:0/3001783539 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7ff4740abc90 con 0x7ff4780b21d0 2024-04-24T23:49:29.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.232+0000 7ff47d2ea640 1 --2- 172.21.15.16:0/3001783539 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff45c06be30 0x7ff45c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.232+0000 7ff47d2ea640 1 --2- 172.21.15.16:0/3001783539 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff45c06be30 0x7ff45c06e2f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff47813ec60 tx=0x7ff468008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:29.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.236+0000 7ff46e7fc640 1 -- 172.21.15.16:0/3001783539 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff47407a420 con 0x7ff4780b21d0 2024-04-24T23:49:29.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.416+0000 7ff47fd4f640 1 -- 172.21.15.16:0/3001783539 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff478140e90 con 0x7ff4780b21d0 2024-04-24T23:49:29.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.416+0000 7ff46e7fc640 1 -- 172.21.15.16:0/3001783539 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) ==== 103+0+11 (secure 0 0 0) 0x7ff474079d20 con 0x7ff4780b21d0 2024-04-24T23:49:29.416 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:29.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.418+0000 7ff47fd4f640 1 -- 172.21.15.16:0/3001783539 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff45c06be30 msgr2=0x7ff45c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.418+0000 7ff47fd4f640 1 --2- 172.21.15.16:0/3001783539 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff45c06be30 0x7ff45c06e2f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff47813ec60 tx=0x7ff468008040 comp rx=0 tx=0).stop 2024-04-24T23:49:29.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.418+0000 7ff47fd4f640 1 -- 172.21.15.16:0/3001783539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4780b21d0 msgr2=0x7ff478140410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.418+0000 7ff47fd4f640 1 --2- 172.21.15.16:0/3001783539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4780b21d0 0x7ff478140410 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff474009870 tx=0x7ff4740046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.418+0000 7ff47fd4f640 1 -- 172.21.15.16:0/3001783539 shutdown_connections 2024-04-24T23:49:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.418+0000 7ff47fd4f640 1 --2- 172.21.15.16:0/3001783539 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff45c06be30 0x7ff45c06e2f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.418+0000 7ff47fd4f640 1 --2- 172.21.15.16:0/3001783539 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4781478d0 0x7ff47815e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.418+0000 7ff47fd4f640 1 --2- 172.21.15.16:0/3001783539 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4780d2090 0x7ff478140950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.418+0000 7ff47fd4f640 1 --2- 172.21.15.16:0/3001783539 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4780b21d0 0x7ff478140410 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.418+0000 7ff47fd4f640 1 -- 172.21.15.16:0/3001783539 >> 172.21.15.16:0/3001783539 conn(0x7ff47806b2c0 msgr2=0x7ff4780b6430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.418+0000 7ff47fd4f640 1 -- 172.21.15.16:0/3001783539 shutdown_connections 2024-04-24T23:49:29.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.418+0000 7ff47fd4f640 1 -- 172.21.15.16:0/3001783539 wait complete. 2024-04-24T23:49:29.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:49:29.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.529+0000 7f144b67d640 1 Processor -- start 2024-04-24T23:49:29.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.530+0000 7f144b67d640 1 -- start start 2024-04-24T23:49:29.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.530+0000 7f144b67d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14440b2bb0 0x7f14440cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.530+0000 7f144b67d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14440b3390 con 0x7f144415e020 2024-04-24T23:49:29.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.530+0000 7f144b67d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f144406bfc0 con 0x7f14440d1fa0 2024-04-24T23:49:29.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.530+0000 7f144b67d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f144406c2a0 con 0x7f14440b2bb0 2024-04-24T23:49:29.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f1449419640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f144415e020 0x7f14440cf270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38518/0 (socket says 172.21.15.16:38518) 2024-04-24T23:49:29.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f1449419640 1 -- 172.21.15.16:0/2826870468 learned_addr learned my addr 172.21.15.16:0/2826870468 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:29.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f1448c18640 1 --2- 172.21.15.16:0/2826870468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14440b2bb0 0x7f14440cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 -- 172.21.15.16:0/2826870468 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 333410047 0 0) 0x7f14440b3390 con 0x7f144415e020 2024-04-24T23:49:29.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 -- 172.21.15.16:0/2826870468 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1430003520 con 0x7f144415e020 2024-04-24T23:49:29.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 -- 172.21.15.16:0/2826870468 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3649368189 0 0) 0x7f1430003520 con 0x7f144415e020 2024-04-24T23:49:29.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 -- 172.21.15.16:0/2826870468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14440b2bb0 msgr2=0x7f14440cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:29.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 --2- 172.21.15.16:0/2826870468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14440b2bb0 0x7f14440cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 -- 172.21.15.16:0/2826870468 >> v1:172.21.15.156:6789/0 conn(0x7f14440d1fa0 legacy=0x7f14440d2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 -- 172.21.15.16:0/2826870468 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144406c580 con 0x7f144415e020 2024-04-24T23:49:29.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 -- 172.21.15.16:0/2826870468 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f1440001670 con 0x7f144415e020 2024-04-24T23:49:29.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 -- 172.21.15.16:0/2826870468 >> v1:172.21.15.16:6789/0 conn(0x7f144415e020 legacy=0x7f14440cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 --2- 172.21.15.16:0/2826870468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14300039e0 0x7f1430003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 --2- 172.21.15.16:0/2826870468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14440b2bb0 0x7f1430004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 --2- 172.21.15.16:0/2826870468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1430004d40 0x7f14300051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 -- 172.21.15.16:0/2826870468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f144406c2a0 con 0x7f14300039e0 2024-04-24T23:49:29.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 -- 172.21.15.16:0/2826870468 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f143000baf0 con 0x7f1430004d40 2024-04-24T23:49:29.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f143bfff640 1 -- 172.21.15.16:0/2826870468 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f143000bcf0 con 0x7f14440b2bb0 2024-04-24T23:49:29.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f1449419640 1 --2- 172.21.15.16:0/2826870468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14300039e0 0x7f1430003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f1448c18640 1 --2- 172.21.15.16:0/2826870468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14440b2bb0 0x7f1430004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.531+0000 7f1449c1a640 1 --2- 172.21.15.16:0/2826870468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1430004d40 0x7f14300051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.532+0000 7f1449419640 1 -- 172.21.15.16:0/2826870468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14440b2bb0 msgr2=0x7f1430004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:29.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.532+0000 7f1449419640 1 --2- 172.21.15.16:0/2826870468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14440b2bb0 0x7f1430004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.532+0000 7f1449419640 1 -- 172.21.15.16:0/2826870468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1430004d40 msgr2=0x7f14300051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:29.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.532+0000 7f1449419640 1 --2- 172.21.15.16:0/2826870468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1430004d40 0x7f14300051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.532+0000 7f1449419640 1 -- 172.21.15.16:0/2826870468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1430003520 con 0x7f14300039e0 2024-04-24T23:49:29.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.532+0000 7f1449419640 1 --2- 172.21.15.16:0/2826870468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14300039e0 0x7f1430003e00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f144000c210 tx=0x7f1430005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3447df73e5da4a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:29.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.532+0000 7f143bfff640 1 -- 172.21.15.16:0/2826870468 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1440007d30 con 0x7f14300039e0 2024-04-24T23:49:29.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.533+0000 7f143bfff640 1 -- 172.21.15.16:0/2826870468 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1440005af0 con 0x7f14300039e0 2024-04-24T23:49:29.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.533+0000 7f143bfff640 1 -- 172.21.15.16:0/2826870468 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1440007070 con 0x7f14300039e0 2024-04-24T23:49:29.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.533+0000 7f144b67d640 1 -- 172.21.15.16:0/2826870468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14300039e0 msgr2=0x7f1430003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.533+0000 7f144b67d640 1 --2- 172.21.15.16:0/2826870468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14300039e0 0x7f1430003e00 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f144000c210 tx=0x7f1430005830 comp rx=0 tx=0).stop 2024-04-24T23:49:29.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.533+0000 7f1449c1a640 1 -- 172.21.15.16:0/2826870468 reap_dead start 2024-04-24T23:49:29.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.533+0000 7f144b67d640 1 -- 172.21.15.16:0/2826870468 shutdown_connections 2024-04-24T23:49:29.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.533+0000 7f144b67d640 1 -- 172.21.15.16:0/2826870468 >> 172.21.15.16:0/2826870468 conn(0x7f14440d3e10 msgr2=0x7f14440d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:29.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.533+0000 7f144b67d640 1 -- 172.21.15.16:0/2826870468 shutdown_connections 2024-04-24T23:49:29.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.533+0000 7f144b67d640 1 -- 172.21.15.16:0/2826870468 wait complete. 2024-04-24T23:49:29.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.534+0000 7f144b67d640 1 Processor -- start 2024-04-24T23:49:29.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.534+0000 7f144b67d640 1 -- start start 2024-04-24T23:49:29.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f144b67d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14300039e0 0x7f1444150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f144b67d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14440cda40 0x7f14441e6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f144b67d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14440cdec0 0x7f14441e70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f144b67d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f144415e400 con 0x7f14440cdec0 2024-04-24T23:49:29.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f144b67d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f14440d1a80 con 0x7f14440cda40 2024-04-24T23:49:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f144b67d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f14440cf980 con 0x7f14300039e0 2024-04-24T23:49:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f1449c1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14440cdec0 0x7f14441e70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f1449c1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14440cdec0 0x7f14441e70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49314/0 (socket says 172.21.15.16:49314) 2024-04-24T23:49:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f1449c1a640 1 -- 172.21.15.16:0/1475491375 learned_addr learned my addr 172.21.15.16:0/1475491375 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f1449419640 1 --2- 172.21.15.16:0/1475491375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14300039e0 0x7f1444150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f1448c18640 1 --2- 172.21.15.16:0/1475491375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14440cda40 0x7f14441e6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f1449c1a640 1 -- 172.21.15.16:0/1475491375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14300039e0 msgr2=0x7f1444150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f1449c1a640 1 --2- 172.21.15.16:0/1475491375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14300039e0 0x7f1444150270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f1449c1a640 1 -- 172.21.15.16:0/1475491375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14440cda40 msgr2=0x7f14441e6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.535+0000 7f1449c1a640 1 --2- 172.21.15.16:0/1475491375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14440cda40 0x7f14441e6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.536+0000 7f1449c1a640 1 -- 172.21.15.16:0/1475491375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14441507b0 con 0x7f14440cdec0 2024-04-24T23:49:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.536+0000 7f1449c1a640 1 --2- 172.21.15.16:0/1475491375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14440cdec0 0x7f14441e70d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f14440ce410 tx=0x7f143c00fc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.536+0000 7f1439ffb640 1 -- 172.21.15.16:0/1475491375 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c01a070 con 0x7f14440cdec0 2024-04-24T23:49:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.536+0000 7f1439ffb640 1 -- 172.21.15.16:0/1475491375 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f143c006d60 con 0x7f14440cdec0 2024-04-24T23:49:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.536+0000 7f144b67d640 1 -- 172.21.15.16:0/1475491375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14440cc9a0 con 0x7f14440cdec0 2024-04-24T23:49:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.536+0000 7f1439ffb640 1 -- 172.21.15.16:0/1475491375 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143c003b90 con 0x7f14440cdec0 2024-04-24T23:49:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.536+0000 7f144b67d640 1 -- 172.21.15.16:0/1475491375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14440ccf60 con 0x7f14440cdec0 2024-04-24T23:49:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.537+0000 7f144b67d640 1 -- 172.21.15.16:0/1475491375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f144406dfe0 con 0x7f14440cdec0 2024-04-24T23:49:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.537+0000 7f1439ffb640 1 -- 172.21.15.16:0/1475491375 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f143c01d070 con 0x7f14440cdec0 2024-04-24T23:49:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.538+0000 7f1439ffb640 1 --2- 172.21.15.16:0/1475491375 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f142806be30 0x7f142806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.538+0000 7f1439ffb640 1 -- 172.21.15.16:0/1475491375 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f143c08a5f0 con 0x7f14440cdec0 2024-04-24T23:49:29.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.538+0000 7f1449419640 1 --2- 172.21.15.16:0/1475491375 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f142806be30 0x7f142806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.539+0000 7f1449419640 1 --2- 172.21.15.16:0/1475491375 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f142806be30 0x7f142806e2f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1440032000 tx=0x7f1440034040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:29.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.542+0000 7f1439ffb640 1 -- 172.21.15.16:0/1475491375 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f14441507b0 con 0x7f14440cdec0 2024-04-24T23:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.764+0000 7f144b67d640 1 -- 172.21.15.16:0/1475491375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f14440ced00 con 0x7f14440cdec0 2024-04-24T23:49:29.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.764+0000 7f1439ffb640 1 -- 172.21.15.16:0/1475491375 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5180 (secure 0 0 0) 0x7f143c058610 con 0x7f14440cdec0 2024-04-24T23:49:29.764 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:29.764 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1623032341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1623032341},{"type":"v1","addr":"172.21.15.156:6836","nonce":1623032341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3530738236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3530738236},{"type":"v1","addr":"172.21.15.16:6835","nonce":3530738236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2153063131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2153063131},{"type":"v1","addr":"172.21.15.16:6834","nonce":2153063131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:48:54.852787+0000","modified":"2024-04-24T23:49:17.914730+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/289127714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":289127714},{"type":"v1","addr":"172.21.15.156:6837","nonce":289127714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-24T23:49:29.765 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-04-24T23:49:29.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.766+0000 7f144b67d640 1 -- 172.21.15.16:0/1475491375 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f142806be30 msgr2=0x7f142806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.766+0000 7f144b67d640 1 --2- 172.21.15.16:0/1475491375 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f142806be30 0x7f142806e2f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1440032000 tx=0x7f1440034040 comp rx=0 tx=0).stop 2024-04-24T23:49:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.766+0000 7f144b67d640 1 -- 172.21.15.16:0/1475491375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14440cdec0 msgr2=0x7f14441e70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.766+0000 7f144b67d640 1 --2- 172.21.15.16:0/1475491375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14440cdec0 0x7f14441e70d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f14440ce410 tx=0x7f143c00fc80 comp rx=0 tx=0).stop 2024-04-24T23:49:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.766+0000 7f144b67d640 1 -- 172.21.15.16:0/1475491375 shutdown_connections 2024-04-24T23:49:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.766+0000 7f144b67d640 1 --2- 172.21.15.16:0/1475491375 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f142806be30 0x7f142806e2f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.766+0000 7f144b67d640 1 --2- 172.21.15.16:0/1475491375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14440cdec0 0x7f14441e70d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.766+0000 7f144b67d640 1 --2- 172.21.15.16:0/1475491375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f14440cda40 0x7f14441e6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.766+0000 7f144b67d640 1 --2- 172.21.15.16:0/1475491375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f14300039e0 0x7f1444150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.766+0000 7f144b67d640 1 -- 172.21.15.16:0/1475491375 >> 172.21.15.16:0/1475491375 conn(0x7f14440d3e10 msgr2=0x7f14441605d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:29.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.766+0000 7f144b67d640 1 -- 172.21.15.16:0/1475491375 shutdown_connections 2024-04-24T23:49:29.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.766+0000 7f144b67d640 1 -- 172.21.15.16:0/1475491375 wait complete. 2024-04-24T23:49:29.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:29.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.879+0000 7fea0db14640 1 Processor -- start 2024-04-24T23:49:29.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.879+0000 7fea0db14640 1 -- start start 2024-04-24T23:49:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.880+0000 7fea0db14640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fea0815dfa0 0x7fea080d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.880+0000 7fea0db14640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fea080cdae0 0x7fea080cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.880+0000 7fea0db14640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea080d05a0 con 0x7fea080ce5f0 2024-04-24T23:49:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.880+0000 7fea0db14640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fea080b7360 con 0x7fea0815dfa0 2024-04-24T23:49:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.880+0000 7fea0db14640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fea080cc640 con 0x7fea080cdae0 2024-04-24T23:49:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.880+0000 7fea077fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fea0815dfa0 0x7fea080d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.880+0000 7fea06ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fea080ce5f0 0x7fea080b5f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38534/0 (socket says 172.21.15.16:38534) 2024-04-24T23:49:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.880+0000 7fea06ffd640 1 -- 172.21.15.16:0/520369711 learned_addr learned my addr 172.21.15.16:0/520369711 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.880+0000 7fea077fe640 1 -- 172.21.15.16:0/520369711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fea080cdae0 msgr2=0x7fea080cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.880+0000 7fea07fff640 1 --2- 172.21.15.16:0/520369711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fea080cdae0 0x7fea080cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.880+0000 7fea067fc640 1 -- 172.21.15.16:0/520369711 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3488699199 0 0) 0x7fea080d05a0 con 0x7fea080ce5f0 2024-04-24T23:49:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.880+0000 7fea077fe640 1 --2- 172.21.15.16:0/520369711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fea080cdae0 0x7fea080cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.880+0000 7fea077fe640 1 -- 172.21.15.16:0/520369711 >> v1:172.21.15.16:6789/0 conn(0x7fea080ce5f0 legacy=0x7fea080b5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.880+0000 7fea077fe640 1 -- 172.21.15.16:0/520369711 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea080d0880 con 0x7fea0815dfa0 2024-04-24T23:49:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.881+0000 7fea077fe640 1 --2- 172.21.15.16:0/520369711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea0815dfa0 0x7fea080d0fe0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe9f80097a0 tx=0x7fe9f802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf3dab83d6ab5b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.881+0000 7fea067fc640 1 -- 172.21.15.16:0/520369711 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8037070 con 0x7fea0815dfa0 2024-04-24T23:49:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.881+0000 7fea067fc640 1 -- 172.21.15.16:0/520369711 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9f8032440 con 0x7fea0815dfa0 2024-04-24T23:49:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.881+0000 7fea067fc640 1 -- 172.21.15.16:0/520369711 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9f8032810 con 0x7fea0815dfa0 2024-04-24T23:49:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.882+0000 7fea0db14640 1 -- 172.21.15.16:0/520369711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea0815dfa0 msgr2=0x7fea080d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.882+0000 7fea0db14640 1 --2- 172.21.15.16:0/520369711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea0815dfa0 0x7fea080d0fe0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe9f80097a0 tx=0x7fe9f802a180 comp rx=0 tx=0).stop 2024-04-24T23:49:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.882+0000 7fea0db14640 1 -- 172.21.15.16:0/520369711 shutdown_connections 2024-04-24T23:49:29.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.882+0000 7fea0db14640 1 --2- 172.21.15.16:0/520369711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fea080cdae0 0x7fea080cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.882+0000 7fea0db14640 1 --2- 172.21.15.16:0/520369711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea0815dfa0 0x7fea080d0fe0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.882+0000 7fea0db14640 1 -- 172.21.15.16:0/520369711 >> 172.21.15.16:0/520369711 conn(0x7fea0806b4d0 msgr2=0x7fea080b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.882+0000 7fea0db14640 1 -- 172.21.15.16:0/520369711 shutdown_connections 2024-04-24T23:49:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.882+0000 7fea0db14640 1 -- 172.21.15.16:0/520369711 wait complete. 2024-04-24T23:49:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.883+0000 7fea0db14640 1 Processor -- start 2024-04-24T23:49:29.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.883+0000 7fea0db14640 1 -- start start 2024-04-24T23:49:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea0db14640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fea080cdae0 0x7fea080cb820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea0db14640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea080ce5f0 0x7fea08158730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea0db14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea0815dfa0 0x7fea081f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea0db14640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fea0815e380 con 0x7fea0815dfa0 2024-04-24T23:49:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea0db14640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fea080cc2d0 con 0x7fea080ce5f0 2024-04-24T23:49:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea0db14640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fea080d1520 con 0x7fea080cdae0 2024-04-24T23:49:29.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea0815dfa0 0x7fea081f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea07fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea0815dfa0 0x7fea081f3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49340/0 (socket says 172.21.15.16:49340) 2024-04-24T23:49:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea07fff640 1 -- 172.21.15.16:0/2926300207 learned_addr learned my addr 172.21.15.16:0/2926300207 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea06ffd640 1 --2- 172.21.15.16:0/2926300207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea080ce5f0 0x7fea08158730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea077fe640 1 --2- 172.21.15.16:0/2926300207 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fea080cdae0 0x7fea080cb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea07fff640 1 -- 172.21.15.16:0/2926300207 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fea080cdae0 msgr2=0x7fea080cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea07fff640 1 --2- 172.21.15.16:0/2926300207 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fea080cdae0 0x7fea080cb820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea07fff640 1 -- 172.21.15.16:0/2926300207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea080ce5f0 msgr2=0x7fea08158730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea07fff640 1 --2- 172.21.15.16:0/2926300207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea080ce5f0 0x7fea08158730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.884+0000 7fea07fff640 1 -- 172.21.15.16:0/2926300207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea081f4660 con 0x7fea0815dfa0 2024-04-24T23:49:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.885+0000 7fea07fff640 1 --2- 172.21.15.16:0/2926300207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea0815dfa0 0x7fea081f3e10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fea0000cd60 tx=0x7fea0000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.885+0000 7fe9effff640 1 -- 172.21.15.16:0/2926300207 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00003b90 con 0x7fea0815dfa0 2024-04-24T23:49:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.885+0000 7fe9effff640 1 -- 172.21.15.16:0/2926300207 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea00003d50 con 0x7fea0815dfa0 2024-04-24T23:49:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.885+0000 7fe9effff640 1 -- 172.21.15.16:0/2926300207 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea00004260 con 0x7fea0815dfa0 2024-04-24T23:49:29.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.885+0000 7fea0db14640 1 -- 172.21.15.16:0/2926300207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea081f4940 con 0x7fea0815dfa0 2024-04-24T23:49:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.885+0000 7fea0db14640 1 -- 172.21.15.16:0/2926300207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea08158c70 con 0x7fea0815dfa0 2024-04-24T23:49:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.886+0000 7fea0db14640 1 -- 172.21.15.16:0/2926300207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea081f4b10 con 0x7fea0815dfa0 2024-04-24T23:49:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.886+0000 7fe9effff640 1 -- 172.21.15.16:0/2926300207 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fea00004400 con 0x7fea0815dfa0 2024-04-24T23:49:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.887+0000 7fe9effff640 1 --2- 172.21.15.16:0/2926300207 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe9f406be30 0x7fe9f406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.887+0000 7fe9effff640 1 -- 172.21.15.16:0/2926300207 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7fea0008bb00 con 0x7fea0815dfa0 2024-04-24T23:49:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.887+0000 7fea077fe640 1 --2- 172.21.15.16:0/2926300207 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe9f406be30 0x7fe9f406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:29.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.888+0000 7fea077fe640 1 --2- 172.21.15.16:0/2926300207 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe9f406be30 0x7fe9f406e2f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8034000 tx=0x7fe9f802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:29.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:29.891+0000 7fe9effff640 1 -- 172.21.15.16:0/2926300207 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fea00059b20 con 0x7fea0815dfa0 2024-04-24T23:49:30.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.074+0000 7fea0db14640 1 -- 172.21.15.16:0/2926300207 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fea0806bea0 con 0x7fea0815dfa0 2024-04-24T23:49:30.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.075+0000 7fe9effff640 1 -- 172.21.15.16:0/2926300207 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) ==== 74+0+18502 (secure 0 0 0) 0x7fea0005de20 con 0x7fea0815dfa0 2024-04-24T23:49:30.075 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:30.075 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:48:59.888015+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:30.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.077+0000 7fea0db14640 1 -- 172.21.15.16:0/2926300207 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe9f406be30 msgr2=0x7fe9f406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.077+0000 7fea0db14640 1 --2- 172.21.15.16:0/2926300207 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe9f406be30 0x7fe9f406e2f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8034000 tx=0x7fe9f802e000 comp rx=0 tx=0).stop 2024-04-24T23:49:30.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.077+0000 7fea0db14640 1 -- 172.21.15.16:0/2926300207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea0815dfa0 msgr2=0x7fea081f3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.077+0000 7fea0db14640 1 --2- 172.21.15.16:0/2926300207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea0815dfa0 0x7fea081f3e10 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fea0000cd60 tx=0x7fea0000a660 comp rx=0 tx=0).stop 2024-04-24T23:49:30.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.077+0000 7fea0db14640 1 -- 172.21.15.16:0/2926300207 shutdown_connections 2024-04-24T23:49:30.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.077+0000 7fea0db14640 1 --2- 172.21.15.16:0/2926300207 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe9f406be30 0x7fe9f406e2f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.077+0000 7fea0db14640 1 --2- 172.21.15.16:0/2926300207 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fea0815dfa0 0x7fea081f3e10 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.077+0000 7fea0db14640 1 --2- 172.21.15.16:0/2926300207 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea080ce5f0 0x7fea08158730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.077+0000 7fea0db14640 1 --2- 172.21.15.16:0/2926300207 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fea080cdae0 0x7fea080cb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.077+0000 7fea0db14640 1 -- 172.21.15.16:0/2926300207 >> 172.21.15.16:0/2926300207 conn(0x7fea0806b4d0 msgr2=0x7fea08160150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:30.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.077+0000 7fea0db14640 1 -- 172.21.15.16:0/2926300207 shutdown_connections 2024-04-24T23:49:30.078 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.077+0000 7fea0db14640 1 -- 172.21.15.16:0/2926300207 wait complete. 2024-04-24T23:49:30.088 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:49:30.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.190+0000 7f95c1f14640 1 Processor -- start 2024-04-24T23:49:30.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.190+0000 7f95c1f14640 1 -- start start 2024-04-24T23:49:30.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.191+0000 7f95c1f14640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95bc0d16c0 0x7f95bc1601f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.191+0000 7f95c1f14640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95bc0705b0 0x7f95bc06faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.191+0000 7f95c1f14640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95bc1607d0 con 0x7f95bc15bcd0 2024-04-24T23:49:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.191+0000 7f95c1f14640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f95bc06f660 con 0x7f95bc0d16c0 2024-04-24T23:49:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.191+0000 7f95c1f14640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f95bc06ed30 con 0x7f95bc0705b0 2024-04-24T23:49:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.191+0000 7f95baffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95bc0d16c0 0x7f95bc1601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.191+0000 7f95bb7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f95bc15bcd0 0x7f95bc0d0fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38548/0 (socket says 172.21.15.16:38548) 2024-04-24T23:49:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.191+0000 7f95bb7fe640 1 -- 172.21.15.16:0/3136975245 learned_addr learned my addr 172.21.15.16:0/3136975245 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.191+0000 7f95bbfff640 1 --2- 172.21.15.16:0/3136975245 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95bc0705b0 0x7f95bc06faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.191+0000 7f95baffd640 1 -- 172.21.15.16:0/3136975245 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95bc0705b0 msgr2=0x7f95bc06faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.191+0000 7f95baffd640 1 --2- 172.21.15.16:0/3136975245 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95bc0705b0 0x7f95bc06faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.191+0000 7f95baffd640 1 -- 172.21.15.16:0/3136975245 >> v1:172.21.15.16:6789/0 conn(0x7f95bc15bcd0 legacy=0x7f95bc0d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.191+0000 7f95baffd640 1 -- 172.21.15.16:0/3136975245 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95bc06f010 con 0x7f95bc0d16c0 2024-04-24T23:49:30.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.192+0000 7f95baffd640 1 --2- 172.21.15.16:0/3136975245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc0d16c0 0x7f95bc1601f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f95b00097a0 tx=0x7f95b002a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9835647d520944b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:30.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.192+0000 7f95ba7fc640 1 -- 172.21.15.16:0/3136975245 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b0037070 con 0x7f95bc0d16c0 2024-04-24T23:49:30.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.192+0000 7f95ba7fc640 1 -- 172.21.15.16:0/3136975245 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95b002adb0 con 0x7f95bc0d16c0 2024-04-24T23:49:30.193 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.192+0000 7f95ba7fc640 1 -- 172.21.15.16:0/3136975245 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b0032660 con 0x7f95bc0d16c0 2024-04-24T23:49:30.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.193+0000 7f95c1f14640 1 -- 172.21.15.16:0/3136975245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc0d16c0 msgr2=0x7f95bc1601f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.193+0000 7f95c1f14640 1 --2- 172.21.15.16:0/3136975245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc0d16c0 0x7f95bc1601f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f95b00097a0 tx=0x7f95b002a060 comp rx=0 tx=0).stop 2024-04-24T23:49:30.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.193+0000 7f95c1f14640 1 -- 172.21.15.16:0/3136975245 shutdown_connections 2024-04-24T23:49:30.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.193+0000 7f95c1f14640 1 --2- 172.21.15.16:0/3136975245 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95bc0705b0 0x7f95bc06faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.193+0000 7f95c1f14640 1 --2- 172.21.15.16:0/3136975245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc0d16c0 0x7f95bc1601f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.193+0000 7f95c1f14640 1 -- 172.21.15.16:0/3136975245 >> 172.21.15.16:0/3136975245 conn(0x7f95bc06b4d0 msgr2=0x7f95bc0b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:30.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.193+0000 7f95c1f14640 1 -- 172.21.15.16:0/3136975245 shutdown_connections 2024-04-24T23:49:30.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.193+0000 7f95c1f14640 1 -- 172.21.15.16:0/3136975245 wait complete. 2024-04-24T23:49:30.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.194+0000 7f95c1f14640 1 Processor -- start 2024-04-24T23:49:30.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.194+0000 7f95c1f14640 1 -- start start 2024-04-24T23:49:30.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95c1f14640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc0705b0 0x7f95bc0b4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95c1f14640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95bc0d16c0 0x7f95bc156c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95c1f14640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95bc15bcd0 0x7f95bc0b61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95c1f14640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc0d09d0 con 0x7f95bc0705b0 2024-04-24T23:49:30.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95c1f14640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f95bc15c0b0 con 0x7f95bc15bcd0 2024-04-24T23:49:30.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95c1f14640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f95bc0d0b50 con 0x7f95bc0d16c0 2024-04-24T23:49:30.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95bb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc0705b0 0x7f95bc0b4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95bb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc0705b0 0x7f95bc0b4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49350/0 (socket says 172.21.15.16:49350) 2024-04-24T23:49:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95bb7fe640 1 -- 172.21.15.16:0/868463341 learned_addr learned my addr 172.21.15.16:0/868463341 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95baffd640 1 --2- 172.21.15.16:0/868463341 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95bc0d16c0 0x7f95bc156c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95bbfff640 1 --2- 172.21.15.16:0/868463341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95bc15bcd0 0x7f95bc0b61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95bb7fe640 1 -- 172.21.15.16:0/868463341 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95bc0d16c0 msgr2=0x7f95bc156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95bb7fe640 1 --2- 172.21.15.16:0/868463341 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95bc0d16c0 0x7f95bc156c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95bb7fe640 1 -- 172.21.15.16:0/868463341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95bc15bcd0 msgr2=0x7f95bc0b61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95bb7fe640 1 --2- 172.21.15.16:0/868463341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95bc15bcd0 0x7f95bc0b61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95bb7fe640 1 -- 172.21.15.16:0/868463341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95bc0b6b40 con 0x7f95bc0705b0 2024-04-24T23:49:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95bb7fe640 1 --2- 172.21.15.16:0/868463341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc0705b0 0x7f95bc0b4960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f95ac00c3f0 tx=0x7f95ac00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95a3fff640 1 -- 172.21.15.16:0/868463341 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac017070 con 0x7f95bc0705b0 2024-04-24T23:49:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.195+0000 7f95a3fff640 1 -- 172.21.15.16:0/868463341 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95ac005ce0 con 0x7f95bc0705b0 2024-04-24T23:49:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.196+0000 7f95c1f14640 1 -- 172.21.15.16:0/868463341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95bc0b6e20 con 0x7f95bc0705b0 2024-04-24T23:49:30.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.196+0000 7f95a3fff640 1 -- 172.21.15.16:0/868463341 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac0072f0 con 0x7f95bc0705b0 2024-04-24T23:49:30.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.196+0000 7f95c1f14640 1 -- 172.21.15.16:0/868463341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95bc0b73e0 con 0x7f95bc0705b0 2024-04-24T23:49:30.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.196+0000 7f95c1f14640 1 -- 172.21.15.16:0/868463341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95bc06bea0 con 0x7f95bc0705b0 2024-04-24T23:49:30.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.197+0000 7f95a3fff640 1 -- 172.21.15.16:0/868463341 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f95ac01a070 con 0x7f95bc0705b0 2024-04-24T23:49:30.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.197+0000 7f95a3fff640 1 --2- 172.21.15.16:0/868463341 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f95a806be30 0x7f95a806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.198+0000 7f95a3fff640 1 -- 172.21.15.16:0/868463341 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f95ac08a470 con 0x7f95bc0705b0 2024-04-24T23:49:30.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.198+0000 7f95baffd640 1 --2- 172.21.15.16:0/868463341 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f95a806be30 0x7f95a806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.198+0000 7f95baffd640 1 --2- 172.21.15.16:0/868463341 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f95a806be30 0x7f95a806e2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f95b0034000 tx=0x7f95b002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:30.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.202+0000 7f95a3fff640 1 -- 172.21.15.16:0/868463341 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f95ac058b70 con 0x7f95bc0705b0 2024-04-24T23:49:30.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.421+0000 7f95c1f14640 1 -- 172.21.15.16:0/868463341 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95bc04f5e0 con 0x7f95bc0705b0 2024-04-24T23:49:30.422 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.422+0000 7f95a3fff640 1 -- 172.21.15.16:0/868463341 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5180 (secure 0 0 0) 0x7f95ac058470 con 0x7f95bc0705b0 2024-04-24T23:49:30.422 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:30.422 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1623032341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1623032341},{"type":"v1","addr":"172.21.15.156:6836","nonce":1623032341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3530738236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3530738236},{"type":"v1","addr":"172.21.15.16:6835","nonce":3530738236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2153063131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2153063131},{"type":"v1","addr":"172.21.15.16:6834","nonce":2153063131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:48:54.852787+0000","modified":"2024-04-24T23:49:17.914730+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/289127714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":289127714},{"type":"v1","addr":"172.21.15.156:6837","nonce":289127714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-24T23:49:30.423 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-04-24T23:49:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.424+0000 7f95c1f14640 1 -- 172.21.15.16:0/868463341 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f95a806be30 msgr2=0x7f95a806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.424+0000 7f95c1f14640 1 --2- 172.21.15.16:0/868463341 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f95a806be30 0x7f95a806e2f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f95b0034000 tx=0x7f95b002e000 comp rx=0 tx=0).stop 2024-04-24T23:49:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.424+0000 7f95c1f14640 1 -- 172.21.15.16:0/868463341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc0705b0 msgr2=0x7f95bc0b4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.424+0000 7f95c1f14640 1 --2- 172.21.15.16:0/868463341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc0705b0 0x7f95bc0b4960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f95ac00c3f0 tx=0x7f95ac00c8b0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.424+0000 7f95c1f14640 1 -- 172.21.15.16:0/868463341 shutdown_connections 2024-04-24T23:49:30.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.424+0000 7f95c1f14640 1 --2- 172.21.15.16:0/868463341 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f95a806be30 0x7f95a806e2f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.424+0000 7f95c1f14640 1 --2- 172.21.15.16:0/868463341 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95bc15bcd0 0x7f95bc0b61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.424+0000 7f95c1f14640 1 --2- 172.21.15.16:0/868463341 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95bc0d16c0 0x7f95bc156c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.424+0000 7f95c1f14640 1 --2- 172.21.15.16:0/868463341 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95bc0705b0 0x7f95bc0b4960 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.424+0000 7f95c1f14640 1 -- 172.21.15.16:0/868463341 >> 172.21.15.16:0/868463341 conn(0x7f95bc06b4d0 msgr2=0x7f95bc15e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.424+0000 7f95c1f14640 1 -- 172.21.15.16:0/868463341 shutdown_connections 2024-04-24T23:49:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.424+0000 7f95c1f14640 1 -- 172.21.15.16:0/868463341 wait complete. 2024-04-24T23:49:30.837 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T23:48:54.852787+0000', 'modified': '2024-04-24T23:49:17.914730+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4294}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4294': {'gid': 4294, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.156:6837/289127714', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6835', 'nonce': 289127714}, {'type': 'v1', 'addr': '172.21.15.156:6837', 'nonce': 289127714}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4294, 'qdb_cluster': [4294]} 2024-04-24T23:49:30.838 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T23:49:30.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-24T23:49:30.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.941+0000 7fabba829640 1 Processor -- start 2024-04-24T23:49:30.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.941+0000 7fabba829640 1 -- start start 2024-04-24T23:49:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.942+0000 7fabba829640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabb40ced90 0x7fabb4166d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.942+0000 7fabba829640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb406ea50 con 0x7fabb4144b20 2024-04-24T23:49:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.942+0000 7fabba829640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabb415fe90 con 0x7fabb41457b0 2024-04-24T23:49:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.942+0000 7fabba829640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fabb4160170 con 0x7fabb40ced90 2024-04-24T23:49:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.942+0000 7fabb3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fabb4144b20 0x7fabb4144f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38550/0 (socket says 172.21.15.16:38550) 2024-04-24T23:49:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.942+0000 7fabb3fff640 1 -- 172.21.15.16:0/3767054627 learned_addr learned my addr 172.21.15.16:0/3767054627 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.942+0000 7fabb2ffd640 1 -- 172.21.15.16:0/3767054627 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4208763429 0 0) 0x7fabb406ea50 con 0x7fabb4144b20 2024-04-24T23:49:30.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.942+0000 7fabb2ffd640 1 -- 172.21.15.16:0/3767054627 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab9c003520 con 0x7fabb4144b20 2024-04-24T23:49:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.942+0000 7fabb8dc6640 1 --2- 172.21.15.16:0/3767054627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabb40ced90 0x7fabb4166d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb2ffd640 1 -- 172.21.15.16:0/3767054627 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 331371662 0 0) 0x7fab9c003520 con 0x7fabb4144b20 2024-04-24T23:49:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb2ffd640 1 -- 172.21.15.16:0/3767054627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabb40ced90 msgr2=0x7fabb4166d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb2ffd640 1 --2- 172.21.15.16:0/3767054627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabb40ced90 0x7fabb4166d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb2ffd640 1 -- 172.21.15.16:0/3767054627 >> v1:172.21.15.156:6789/0 conn(0x7fabb41457b0 legacy=0x7fabb40ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb2ffd640 1 -- 172.21.15.16:0/3767054627 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb4160450 con 0x7fabb4144b20 2024-04-24T23:49:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb2ffd640 1 -- 172.21.15.16:0/3767054627 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7faba8001670 con 0x7fabb4144b20 2024-04-24T23:49:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb2ffd640 1 -- 172.21.15.16:0/3767054627 >> v1:172.21.15.16:6789/0 conn(0x7fabb4144b20 legacy=0x7fabb4144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb2ffd640 1 --2- 172.21.15.16:0/3767054627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab9c0039e0 0x7fab9c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb2ffd640 1 --2- 172.21.15.16:0/3767054627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab9c004800 0x7fab9c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb2ffd640 1 --2- 172.21.15.16:0/3767054627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabb40ced90 0x7fab9c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb2ffd640 1 -- 172.21.15.16:0/3767054627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabb4160170 con 0x7fab9c0039e0 2024-04-24T23:49:30.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb2ffd640 1 -- 172.21.15.16:0/3767054627 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fab9c005520 con 0x7fab9c004800 2024-04-24T23:49:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb2ffd640 1 -- 172.21.15.16:0/3767054627 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fab9c00bf00 con 0x7fabb40ced90 2024-04-24T23:49:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb3fff640 1 --2- 172.21.15.16:0/3767054627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab9c0039e0 0x7fab9c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb37fe640 1 --2- 172.21.15.16:0/3767054627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab9c004800 0x7fab9c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb8dc6640 1 --2- 172.21.15.16:0/3767054627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabb40ced90 0x7fab9c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb3fff640 1 -- 172.21.15.16:0/3767054627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabb40ced90 msgr2=0x7fab9c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb3fff640 1 --2- 172.21.15.16:0/3767054627 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabb40ced90 0x7fab9c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb3fff640 1 -- 172.21.15.16:0/3767054627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab9c004800 msgr2=0x7fab9c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb3fff640 1 --2- 172.21.15.16:0/3767054627 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab9c004800 0x7fab9c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.943+0000 7fabb3fff640 1 -- 172.21.15.16:0/3767054627 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab9c003520 con 0x7fab9c0039e0 2024-04-24T23:49:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.944+0000 7fabb3fff640 1 --2- 172.21.15.16:0/3767054627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab9c0039e0 0x7fab9c003e00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7faba800c240 tx=0x7fab9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55cf52f9e59892a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.944+0000 7fabb2ffd640 1 -- 172.21.15.16:0/3767054627 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba8007d10 con 0x7fab9c0039e0 2024-04-24T23:49:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.944+0000 7fabb2ffd640 1 -- 172.21.15.16:0/3767054627 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba8005ad0 con 0x7fab9c0039e0 2024-04-24T23:49:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.944+0000 7fabb2ffd640 1 -- 172.21.15.16:0/3767054627 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba8007070 con 0x7fab9c0039e0 2024-04-24T23:49:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.944+0000 7fabba829640 1 -- 172.21.15.16:0/3767054627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab9c0039e0 msgr2=0x7fab9c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.944+0000 7fabba829640 1 --2- 172.21.15.16:0/3767054627 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab9c0039e0 0x7fab9c003e00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7faba800c240 tx=0x7fab9c005830 comp rx=0 tx=0).stop 2024-04-24T23:49:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.944+0000 7fabb8dc6640 1 -- 172.21.15.16:0/3767054627 reap_dead start 2024-04-24T23:49:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.944+0000 7fabba829640 1 -- 172.21.15.16:0/3767054627 shutdown_connections 2024-04-24T23:49:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.944+0000 7fabba829640 1 -- 172.21.15.16:0/3767054627 >> 172.21.15.16:0/3767054627 conn(0x7fabb40b2000 msgr2=0x7fabb40cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.945+0000 7fabba829640 1 -- 172.21.15.16:0/3767054627 shutdown_connections 2024-04-24T23:49:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.945+0000 7fabba829640 1 -- 172.21.15.16:0/3767054627 wait complete. 2024-04-24T23:49:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.946+0000 7fabba829640 1 Processor -- start 2024-04-24T23:49:30.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.946+0000 7fabba829640 1 -- start start 2024-04-24T23:49:30.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabba829640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab9c0039e0 0x7fabb40a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabba829640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb41f7db0 0x7fabb41f7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabba829640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabb413fb10 0x7fabb413ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabba829640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabb40b3390 con 0x7fabb41f7db0 2024-04-24T23:49:30.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabba829640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fabb406d4b0 con 0x7fab9c0039e0 2024-04-24T23:49:30.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabba829640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fabb415f990 con 0x7fabb413fb10 2024-04-24T23:49:30.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb41f7db0 0x7fabb41f7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb41f7db0 0x7fabb41f7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49370/0 (socket says 172.21.15.16:49370) 2024-04-24T23:49:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb37fe640 1 -- 172.21.15.16:0/1919441825 learned_addr learned my addr 172.21.15.16:0/1919441825 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb8dc6640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabb413fb10 0x7fabb413ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb3fff640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab9c0039e0 0x7fabb40a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb37fe640 1 -- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabb413fb10 msgr2=0x7fabb413ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb37fe640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabb413fb10 0x7fabb413ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb37fe640 1 -- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab9c0039e0 msgr2=0x7fabb40a94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb37fe640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab9c0039e0 0x7fabb40a94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb37fe640 1 -- 172.21.15.16:0/1919441825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabb40cd090 con 0x7fabb41f7db0 2024-04-24T23:49:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb3fff640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab9c0039e0 0x7fabb40a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb37fe640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb41f7db0 0x7fabb41f7480 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7faba400f850 tx=0x7faba400fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb0ff9640 1 -- 172.21.15.16:0/1919441825 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba400fe40 con 0x7fabb41f7db0 2024-04-24T23:49:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb0ff9640 1 -- 172.21.15.16:0/1919441825 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba4018070 con 0x7fabb41f7db0 2024-04-24T23:49:30.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabb0ff9640 1 -- 172.21.15.16:0/1919441825 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba400d730 con 0x7fabb41f7db0 2024-04-24T23:49:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabba829640 1 -- 172.21.15.16:0/1919441825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabb40cd370 con 0x7fabb41f7db0 2024-04-24T23:49:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.947+0000 7fabba829640 1 -- 172.21.15.16:0/1919441825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabb40cd930 con 0x7fabb41f7db0 2024-04-24T23:49:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.949+0000 7fabb0ff9640 1 -- 172.21.15.16:0/1919441825 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7faba400d8d0 con 0x7fabb41f7db0 2024-04-24T23:49:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.949+0000 7fabb0ff9640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7faba006be30 0x7faba006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.950+0000 7fabb0ff9640 1 -- 172.21.15.16:0/1919441825 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7faba408b010 con 0x7fabb41f7db0 2024-04-24T23:49:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.950+0000 7fabb3fff640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7faba006be30 0x7faba006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.950+0000 7fabbd60c740 1 Processor -- start 2024-04-24T23:49:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.950+0000 7fabb3fff640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7faba006be30 0x7faba006e2f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faba8033000 tx=0x7faba802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.950+0000 7fabbd60c740 1 -- start start 2024-04-24T23:49:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabbd60c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc73b60 0x56055dc264f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabbd60c740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 0x56055dc56000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabbd60c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc56540 0x56055dc2b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabbd60c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56055db8ef00 con 0x56055dc73b60 2024-04-24T23:49:30.960 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabbd60c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56055db37750 con 0x56055dc56540 2024-04-24T23:49:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabbd60c740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x56055db8aec0 con 0x56055dc56c80 2024-04-24T23:49:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabb3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc73b60 0x56055dc264f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabb3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc73b60 0x56055dc264f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49378/0 (socket says 172.21.15.16:49378) 2024-04-24T23:49:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabb3fff640 1 -- 172.21.15.16:0/4275550621 learned_addr learned my addr 172.21.15.16:0/4275550621 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabb37fe640 1 --2- 172.21.15.16:0/4275550621 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 0x56055dc56000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabb8dc6640 1 --2- 172.21.15.16:0/4275550621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc56540 0x56055dc2b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabb3fff640 1 -- 172.21.15.16:0/4275550621 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 msgr2=0x56055dc56000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabb3fff640 1 --2- 172.21.15.16:0/4275550621 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 0x56055dc56000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabb3fff640 1 -- 172.21.15.16:0/4275550621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc56540 msgr2=0x56055dc2b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabb3fff640 1 --2- 172.21.15.16:0/4275550621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc56540 0x56055dc2b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabb3fff640 1 -- 172.21.15.16:0/4275550621 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055d8a32d0 con 0x56055dc73b60 2024-04-24T23:49:30.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabb8dc6640 1 --2- 172.21.15.16:0/4275550621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc56540 0x56055dc2b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.951+0000 7fabb37fe640 1 --2- 172.21.15.16:0/4275550621 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 0x56055dc56000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.952+0000 7fabb3fff640 1 --2- 172.21.15.16:0/4275550621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc73b60 0x56055dc264f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7faba8001530 tx=0x7faba8009030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97cc3f09654e7c7f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.952+0000 7fab99ffb640 1 -- 172.21.15.16:0/4275550621 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba8042040 con 0x56055dc73b60 2024-04-24T23:49:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.952+0000 7fab99ffb640 1 -- 172.21.15.16:0/4275550621 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba8007070 con 0x56055dc73b60 2024-04-24T23:49:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.952+0000 7fab99ffb640 1 -- 172.21.15.16:0/4275550621 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba8007420 con 0x56055dc73b60 2024-04-24T23:49:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.952+0000 7fabbd60c740 1 -- 172.21.15.16:0/4275550621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc73b60 msgr2=0x56055dc264f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.952+0000 7fabbd60c740 1 --2- 172.21.15.16:0/4275550621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc73b60 0x56055dc264f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7faba8001530 tx=0x7faba8009030 comp rx=0 tx=0).stop 2024-04-24T23:49:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.952+0000 7fabbd60c740 1 -- 172.21.15.16:0/4275550621 shutdown_connections 2024-04-24T23:49:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.952+0000 7fabbd60c740 1 --2- 172.21.15.16:0/4275550621 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc56540 0x56055dc2b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.952+0000 7fabbd60c740 1 --2- 172.21.15.16:0/4275550621 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 0x56055dc56000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.952+0000 7fabbd60c740 1 --2- 172.21.15.16:0/4275550621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc73b60 0x56055dc264f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.952+0000 7fabbd60c740 1 -- 172.21.15.16:0/4275550621 >> 172.21.15.16:0/4275550621 conn(0x56055dc28760 msgr2=0x56055dc71000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:30.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.952+0000 7fabbd60c740 1 -- 172.21.15.16:0/4275550621 shutdown_connections 2024-04-24T23:49:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.952+0000 7fabbd60c740 1 -- 172.21.15.16:0/4275550621 wait complete. 2024-04-24T23:49:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.953+0000 7fabbd60c740 1 -- start start 2024-04-24T23:49:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.953+0000 7fabbd60c740 1 Processor -- start 2024-04-24T23:49:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabbd60c740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 0x56055dc45950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabbd60c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc73b60 0x56055dc59190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabbd60c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc57980 0x56055dc67cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabbd60c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56055dc2a910 con 0x56055dc73b60 2024-04-24T23:49:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabbd60c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56055db4acf0 con 0x56055dc57980 2024-04-24T23:49:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabbd60c740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x56055dc29d00 con 0x56055dc56c80 2024-04-24T23:49:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabbd60c740 10 client.0 _ll_register_callbacks cb 0x56055db4a6e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:49:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabb37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc73b60 0x56055dc59190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabb37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc73b60 0x56055dc59190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49382/0 (socket says 172.21.15.16:49382) 2024-04-24T23:49:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabb37fe640 1 -- 172.21.15.16:0/494395524 learned_addr learned my addr 172.21.15.16:0/494395524 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:30.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabb3fff640 1 --2- 172.21.15.16:0/494395524 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 0x56055dc45950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabb8dc6640 1 --2- 172.21.15.16:0/494395524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc57980 0x56055dc67cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabb37fe640 1 -- 172.21.15.16:0/494395524 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 msgr2=0x56055dc45950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabb37fe640 1 --2- 172.21.15.16:0/494395524 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 0x56055dc45950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabb37fe640 1 -- 172.21.15.16:0/494395524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc57980 msgr2=0x56055dc67cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabb37fe640 1 --2- 172.21.15.16:0/494395524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc57980 0x56055dc67cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.954+0000 7fabb37fe640 1 -- 172.21.15.16:0/494395524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56055d96d6b0 con 0x56055dc73b60 2024-04-24T23:49:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fabb37fe640 1 --2- 172.21.15.16:0/494395524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc73b60 0x56055dc59190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faba4061630 tx=0x7faba4059740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ba296553005ae3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fab92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-24T23:49:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fab92ffd640 1 -- 172.21.15.16:0/494395524 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba4086070 con 0x56055dc73b60 2024-04-24T23:49:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fab92ffd640 1 -- 172.21.15.16:0/494395524 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7faba400dbc0 con 0x56055dc73b60 2024-04-24T23:49:30.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fabbd60c740 1 -- 172.21.15.16:0/494395524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faba8007070 con 0x56055dc73b60 2024-04-24T23:49:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fab92ffd640 1 -- 172.21.15.16:0/494395524 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7faba4061b60 con 0x56055dc73b60 2024-04-24T23:49:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fabbd60c740 10 client.4821 fetch_fsmap learned FSMap version 8 2024-04-24T23:49:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fabbd60c740 1 -- 172.21.15.16:0/494395524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x56055dc58c70 con 0x56055dc73b60 2024-04-24T23:49:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fab92ffd640 1 -- 172.21.15.16:0/494395524 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba8007070 con 0x56055dc73b60 2024-04-24T23:49:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fab92ffd640 1 -- 172.21.15.16:0/494395524 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7faba4082b60 con 0x56055dc73b60 2024-04-24T23:49:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fabbd60c740 10 client.4821 fetch_fsmap finished waiting for FSMap version 8 2024-04-24T23:49:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fabbd60c740 10 client.4821 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T23:49:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fabbd60c740 20 client.4821 populate_metadata read hostname 'smithi016' 2024-04-24T23:49:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fabbd60c740 1 --2- 172.21.15.16:0/494395524 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x56055dc68380 0x56055dc71680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fabbd60c740 4 client.4821 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:49:30.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fabbd60c740 1 -- 172.21.15.16:0/494395524 --> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56055db9c440 con 0x56055dc68380 2024-04-24T23:49:30.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.955+0000 7fabb3fff640 1 --2- 172.21.15.16:0/494395524 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x56055dc68380 0x56055dc71680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.956+0000 7fab92ffd640 10 client.4821 ms_handle_connect on v2:172.21.15.156:6835/289127714 2024-04-24T23:49:30.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.956+0000 7fabb3fff640 1 --2- 172.21.15.16:0/494395524 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x56055dc68380 0x56055dc71680 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7c304cde9daab1 server_cookie=67977665f3601dfd in_seq=0 out_seq=0 2024-04-24T23:49:30.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.956+0000 7fab92ffd640 1 -- 172.21.15.16:0/494395524 <== mds.0 v2:172.21.15.156:6835/289127714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56055db9c440 con 0x56055dc68380 2024-04-24T23:49:30.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.956+0000 7fab92ffd640 10 client.4821 handle_command_reply: tid=0 2024-04-24T23:49:30.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.957+0000 7fabbd60c740 1 client.4821 shutdown 2024-04-24T23:49:30.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.957+0000 7fabbd60c740 1 -- 172.21.15.16:0/494395524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc73b60 msgr2=0x56055dc59190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.957+0000 7fabbd60c740 1 --2- 172.21.15.16:0/494395524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc73b60 0x56055dc59190 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faba4061630 tx=0x7faba4059740 comp rx=0 tx=0).stop 2024-04-24T23:49:30.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.957+0000 7fabbd60c740 1 -- 172.21.15.16:0/494395524 shutdown_connections 2024-04-24T23:49:30.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.957+0000 7fabbd60c740 1 --2- 172.21.15.16:0/494395524 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc57980 0x56055dc67cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.957+0000 7fabbd60c740 1 --2- 172.21.15.16:0/494395524 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x56055dc68380 0x56055dc71680 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.957+0000 7fabbd60c740 1 --2- 172.21.15.16:0/494395524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc73b60 0x56055dc59190 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.957+0000 7fab92ffd640 0 client.4821 ms_handle_reset on v2:172.21.15.156:6835/289127714 2024-04-24T23:49:30.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.957+0000 7fabbd60c740 1 --2- 172.21.15.16:0/494395524 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 0x56055dc45950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.957+0000 7fabbd60c740 1 -- 172.21.15.16:0/494395524 >> 172.21.15.16:0/494395524 conn(0x56055dc56540 msgr2=0x56055dc74420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:30.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.957+0000 7fabbd60c740 1 -- 172.21.15.16:0/494395524 shutdown_connections 2024-04-24T23:49:30.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.957+0000 7fabbd60c740 1 -- 172.21.15.16:0/494395524 wait complete. 2024-04-24T23:49:30.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.957+0000 7fabbd60c740 20 client.4821 trim_cache size 0 max 16384 2024-04-24T23:49:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.975+0000 7fabbd60c740 1 Processor -- start 2024-04-24T23:49:30.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.975+0000 7fabbd60c740 1 -- start start 2024-04-24T23:49:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabbd60c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc56c80 0x56055dc4fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabbd60c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc57980 0x56055dc70060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabbd60c740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc73b60 0x56055dc705a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabbd60c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fabb4201130 con 0x56055dc57980 2024-04-24T23:49:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabbd60c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fabb41f5e20 con 0x56055dc56c80 2024-04-24T23:49:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabbd60c740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fabb4200ae0 con 0x56055dc73b60 2024-04-24T23:49:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabb37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc57980 0x56055dc70060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabb37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc57980 0x56055dc70060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49386/0 (socket says 172.21.15.16:49386) 2024-04-24T23:49:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabb37fe640 1 -- 172.21.15.16:0/864302189 learned_addr learned my addr 172.21.15.16:0/864302189 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:30.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabb3fff640 1 --2- 172.21.15.16:0/864302189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc56c80 0x56055dc4fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabb8dc6640 1 --2- 172.21.15.16:0/864302189 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc73b60 0x56055dc705a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabb37fe640 1 -- 172.21.15.16:0/864302189 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc73b60 msgr2=0x56055dc705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabb37fe640 1 --2- 172.21.15.16:0/864302189 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc73b60 0x56055dc705a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabb37fe640 1 -- 172.21.15.16:0/864302189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc56c80 msgr2=0x56055dc4fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabb37fe640 1 --2- 172.21.15.16:0/864302189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc56c80 0x56055dc4fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabb37fe640 1 -- 172.21.15.16:0/864302189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56055dc502f0 con 0x56055dc57980 2024-04-24T23:49:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabb3fff640 1 --2- 172.21.15.16:0/864302189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc56c80 0x56055dc4fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.976+0000 7fabb37fe640 1 --2- 172.21.15.16:0/864302189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc57980 0x56055dc70060 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faba40011d0 tx=0x7faba4070000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c245de54e8fad9b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.977+0000 7fab997fa640 1 -- 172.21.15.16:0/864302189 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba4086040 con 0x56055dc57980 2024-04-24T23:49:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.977+0000 7fab997fa640 1 -- 172.21.15.16:0/864302189 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba400de30 con 0x56055dc57980 2024-04-24T23:49:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.977+0000 7fab997fa640 1 -- 172.21.15.16:0/864302189 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba4082b60 con 0x56055dc57980 2024-04-24T23:49:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.977+0000 7fabbd60c740 1 -- 172.21.15.16:0/864302189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc57980 msgr2=0x56055dc70060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.977+0000 7fabbd60c740 1 --2- 172.21.15.16:0/864302189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc57980 0x56055dc70060 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7faba40011d0 tx=0x7faba4070000 comp rx=0 tx=0).stop 2024-04-24T23:49:30.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.977+0000 7fabbd60c740 1 -- 172.21.15.16:0/864302189 shutdown_connections 2024-04-24T23:49:30.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.977+0000 7fabbd60c740 1 --2- 172.21.15.16:0/864302189 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc73b60 0x56055dc705a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.977+0000 7fabbd60c740 1 --2- 172.21.15.16:0/864302189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc57980 0x56055dc70060 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.977+0000 7fabbd60c740 1 --2- 172.21.15.16:0/864302189 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc56c80 0x56055dc4fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.977+0000 7fabbd60c740 1 -- 172.21.15.16:0/864302189 >> 172.21.15.16:0/864302189 conn(0x56055dc56540 msgr2=0x56055dc21690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:30.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.977+0000 7fabbd60c740 1 -- 172.21.15.16:0/864302189 shutdown_connections 2024-04-24T23:49:30.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.977+0000 7fabbd60c740 1 -- 172.21.15.16:0/864302189 wait complete. 2024-04-24T23:49:30.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.978+0000 7fabbd60c740 1 -- start start 2024-04-24T23:49:30.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.978+0000 7fabbd60c740 1 Processor -- start 2024-04-24T23:49:30.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabbd60c740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 0x56055dcea5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabbd60c740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc57980 0x56055dc58e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabbd60c740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc73b60 0x56055dc59380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabbd60c740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56055dc80cc0 con 0x56055dc57980 2024-04-24T23:49:30.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabbd60c740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56055dc68200 con 0x56055dc73b60 2024-04-24T23:49:30.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabbd60c740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x56055dc812e0 con 0x56055dc56c80 2024-04-24T23:49:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabbd60c740 10 client.0 _ll_register_callbacks cb 0x7faba4058b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:49:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabb37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc57980 0x56055dc58e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabb37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc57980 0x56055dc58e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49388/0 (socket says 172.21.15.16:49388) 2024-04-24T23:49:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabb37fe640 1 -- 172.21.15.16:0/3134436147 learned_addr learned my addr 172.21.15.16:0/3134436147 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabb3fff640 1 --2- 172.21.15.16:0/3134436147 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 0x56055dcea5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabb8dc6640 1 --2- 172.21.15.16:0/3134436147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc73b60 0x56055dc59380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabb37fe640 1 -- 172.21.15.16:0/3134436147 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 msgr2=0x56055dcea5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabb37fe640 1 --2- 172.21.15.16:0/3134436147 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 0x56055dcea5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabb37fe640 1 -- 172.21.15.16:0/3134436147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc73b60 msgr2=0x56055dc59380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabb37fe640 1 --2- 172.21.15.16:0/3134436147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc73b60 0x56055dc59380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabb37fe640 1 -- 172.21.15.16:0/3134436147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56055dc4fda0 con 0x56055dc57980 2024-04-24T23:49:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fabb37fe640 1 --2- 172.21.15.16:0/3134436147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc57980 0x56055dc58e40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7faba4058f10 tx=0x7faba40890d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70909ce18a167964 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fab92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-24T23:49:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.979+0000 7fab92ffd640 1 -- 172.21.15.16:0/3134436147 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faba4086050 con 0x56055dc57980 2024-04-24T23:49:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fab92ffd640 1 -- 172.21.15.16:0/3134436147 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7faba4057280 con 0x56055dc57980 2024-04-24T23:49:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fabbd60c740 1 -- 172.21.15.16:0/3134436147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faba400de30 con 0x56055dc57980 2024-04-24T23:49:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fab92ffd640 1 -- 172.21.15.16:0/3134436147 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) ==== 24+0+0 (secure 0 0 0) 0x7faba4068050 con 0x56055dc57980 2024-04-24T23:49:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fabbd60c740 10 client.4827 fetch_fsmap learned FSMap version 8 2024-04-24T23:49:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fabbd60c740 1 -- 172.21.15.16:0/3134436147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) -- 0x56055dc4fda0 con 0x56055dc57980 2024-04-24T23:49:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fab92ffd640 1 -- 172.21.15.16:0/3134436147 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faba400de30 con 0x56055dc57980 2024-04-24T23:49:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fab92ffd640 1 -- 172.21.15.16:0/3134436147 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 8) ==== 3502+0+0 (secure 0 0 0) 0x7faba406b020 con 0x56055dc57980 2024-04-24T23:49:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fabbd60c740 10 client.4827 fetch_fsmap finished waiting for FSMap version 8 2024-04-24T23:49:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fabbd60c740 10 client.4827 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T23:49:30.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fabbd60c740 20 client.4827 populate_metadata read hostname 'smithi016' 2024-04-24T23:49:30.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fabbd60c740 1 --2- 172.21.15.16:0/3134436147 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x56055dc935a0 0x56055dc93980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:30.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fabbd60c740 4 client.4827 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:49:30.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fabbd60c740 1 -- 172.21.15.16:0/3134436147 --> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] -- command(tid 0: {"prefix": "status"}) -- 0x56055dc27bb0 con 0x56055dc935a0 2024-04-24T23:49:30.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fabb3fff640 1 --2- 172.21.15.16:0/3134436147 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x56055dc935a0 0x56055dc93980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:30.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fab92ffd640 10 client.4827 ms_handle_connect on v2:172.21.15.156:6835/289127714 2024-04-24T23:49:30.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.980+0000 7fabb3fff640 1 --2- 172.21.15.16:0/3134436147 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x56055dc935a0 0x56055dc93980 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=490a23eacd50b2cd server_cookie=f5eb53ca81eaa66c in_seq=0 out_seq=0 2024-04-24T23:49:30.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fab92ffd640 1 -- 172.21.15.16:0/3134436147 <== mds.0 v2:172.21.15.156:6835/289127714 1 ==== command_reply(tid 0: 0 ) ==== 8+0+330 (crc 0 0 0) 0x56055dc27bb0 con 0x56055dc935a0 2024-04-24T23:49:30.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fab92ffd640 10 client.4827 handle_command_reply: tid=0 2024-04-24T23:49:30.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fabbd60c740 1 client.4827 shutdown 2024-04-24T23:49:30.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fabbd60c740 1 -- 172.21.15.16:0/3134436147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc57980 msgr2=0x56055dc58e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fabbd60c740 1 --2- 172.21.15.16:0/3134436147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc57980 0x56055dc58e40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7faba4058f10 tx=0x7faba40890d0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fabbd60c740 1 -- 172.21.15.16:0/3134436147 shutdown_connections 2024-04-24T23:49:30.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fabbd60c740 1 --2- 172.21.15.16:0/3134436147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56055dc73b60 0x56055dc59380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fabbd60c740 1 --2- 172.21.15.16:0/3134436147 >> [v2:172.21.15.156:6835/289127714,v1:172.21.15.156:6837/289127714] conn(0x56055dc935a0 0x56055dc93980 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fabbd60c740 1 --2- 172.21.15.16:0/3134436147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56055dc57980 0x56055dc58e40 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fab92ffd640 0 client.4827 ms_handle_reset on v2:172.21.15.156:6835/289127714 2024-04-24T23:49:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fabbd60c740 1 --2- 172.21.15.16:0/3134436147 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56055dc56c80 0x56055dcea5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fabbd60c740 1 -- 172.21.15.16:0/3134436147 >> 172.21.15.16:0/3134436147 conn(0x56055dc56540 msgr2=0x56055dc714f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fabbd60c740 1 -- 172.21.15.16:0/3134436147 shutdown_connections 2024-04-24T23:49:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fabbd60c740 1 -- 172.21.15.16:0/3134436147 wait complete. 2024-04-24T23:49:30.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.981+0000 7fabbd60c740 20 client.4827 trim_cache size 0 max 16384 2024-04-24T23:49:30.987 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:49:30.987 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "f8999c8c-ab57-4aba-8d53-08e19e6ee3d5", 2024-04-24T23:49:30.988 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-04-24T23:49:30.988 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4294, 2024-04-24T23:49:30.988 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-24T23:49:30.988 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-24T23:49:30.988 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-24T23:49:30.988 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 35.491934634000003, 2024-04-24T23:49:30.988 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 8, 2024-04-24T23:49:30.988 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 25, 2024-04-24T23:49:30.988 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 25, 2024-04-24T23:49:30.988 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 42.339733797999997 2024-04-24T23:49:30.988 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:49:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.982+0000 7fabba829640 1 -- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7faba006be30 msgr2=0x7faba006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.982+0000 7fabba829640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7faba006be30 0x7faba006e2f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7faba8033000 tx=0x7faba802e000 comp rx=0 tx=0).stop 2024-04-24T23:49:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.982+0000 7fabba829640 1 -- 172.21.15.16:0/1919441825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb41f7db0 msgr2=0x7fabb41f7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.982+0000 7fabba829640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb41f7db0 0x7fabb41f7480 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7faba400f850 tx=0x7faba400fd10 comp rx=0 tx=0).stop 2024-04-24T23:49:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.982+0000 7fabba829640 1 -- 172.21.15.16:0/1919441825 shutdown_connections 2024-04-24T23:49:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.982+0000 7fabba829640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7faba006be30 0x7faba006e2f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.982+0000 7fabba829640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fabb413fb10 0x7fabb413ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.982+0000 7fabba829640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fabb41f7db0 0x7fabb41f7480 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.982+0000 7fabba829640 1 --2- 172.21.15.16:0/1919441825 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fab9c0039e0 0x7fabb40a94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:30.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.982+0000 7fabba829640 1 -- 172.21.15.16:0/1919441825 >> 172.21.15.16:0/1919441825 conn(0x7fabb40b2000 msgr2=0x7fabb415c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:30.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.982+0000 7fabba829640 1 -- 172.21.15.16:0/1919441825 shutdown_connections 2024-04-24T23:49:30.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:30.982+0000 7fabba829640 1 -- 172.21.15.16:0/1919441825 wait complete. 2024-04-24T23:49:30.992 INFO:teuthology.run_tasks:Running task sequential... 2024-04-24T23:49:31.002 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-24T23:49:31.002 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-24T23:49:31.003 INFO:teuthology.task.print:Enabling mgr modules 2024-04-24T23:49:31.003 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-04-24T23:49:31.012 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-04-24T23:49:31.012 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-04-24T23:49:31.013 INFO:tasks.ceph_fuse:client.0 config is {} 2024-04-24T23:49:31.013 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T23:49:31.013 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T23:49:31.013 INFO:tasks.ceph_fuse:client.2 config is {} 2024-04-24T23:49:31.013 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T23:49:31.013 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-24T23:49:31.013 INFO:tasks.ceph_fuse:client.1 config is {} 2024-04-24T23:49:31.013 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T23:49:31.013 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-24T23:49:31.014 INFO:tasks.ceph_fuse:client.3 config is {} 2024-04-24T23:49:31.014 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T23:49:31.014 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-24T23:49:31.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:31.014 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-24T23:49:31.030 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-24T23:49:31.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:31.030 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-24T23:49:31.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:31.095 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-04-24T23:49:31.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:31.198 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2024-04-24T23:49:31.217 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:31.217 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link delete ceph-brx 2024-04-24T23:49:31.288 INFO:teuthology.orchestra.run.smithi156.stderr:Cannot find device "ceph-brx" 2024-04-24T23:49:31.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:49:31.289 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-04-24T23:49:31.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T23:49:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.392+0000 7f1cedf89640 1 Processor -- start 2024-04-24T23:49:31.392 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.392+0000 7f1cedf89640 1 -- start start 2024-04-24T23:49:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.393+0000 7f1cedf89640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ce815e020 0x7f1ce80cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.393+0000 7f1cedf89640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1ce80b2bb0 0x7f1ce80cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.393+0000 7f1cedf89640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ce80b3390 con 0x7f1ce80cfcf0 2024-04-24T23:49:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.393+0000 7f1cedf89640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f1ce806bfc0 con 0x7f1ce80b2bb0 2024-04-24T23:49:31.393 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.393+0000 7f1cedf89640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1ce806c2a0 con 0x7f1ce815e020 2024-04-24T23:49:31.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.394+0000 7f1ce77fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ce815e020 0x7f1ce80cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:31.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.394+0000 7f1ce6ffd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1ce80b2bb0 0x7f1ce80cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:31.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.394+0000 7f1ce7fff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f1ce80cfcf0 0x7f1ce8161bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:35262/0 (socket says 172.21.15.16:35262) 2024-04-24T23:49:31.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.394+0000 7f1ce7fff640 1 -- 172.21.15.16:0/2104367750 learned_addr learned my addr 172.21.15.16:0/2104367750 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:31.394 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.394+0000 7f1ce67fc640 1 -- 172.21.15.16:0/2104367750 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2566974948 0 0) 0x7f1ce80b3390 con 0x7f1ce80cfcf0 2024-04-24T23:49:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.394+0000 7f1ce67fc640 1 -- 172.21.15.16:0/2104367750 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cd0003520 con 0x7f1ce80cfcf0 2024-04-24T23:49:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.394+0000 7f1ce77fe640 1 -- 172.21.15.16:0/2104367750 >> v2:172.21.15.156:3300/0 conn(0x7f1ce80b2bb0 msgr2=0x7f1ce80cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.394+0000 7f1ce77fe640 1 --2- 172.21.15.16:0/2104367750 >> v2:172.21.15.156:3300/0 conn(0x7f1ce80b2bb0 0x7f1ce80cf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.394+0000 7f1ce77fe640 1 -- 172.21.15.16:0/2104367750 >> v1:172.21.15.156:6789/0 conn(0x7f1ce80cfcf0 legacy=0x7f1ce8161bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.394+0000 7f1ce77fe640 1 -- 172.21.15.16:0/2104367750 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce806c580 con 0x7f1ce815e020 2024-04-24T23:49:31.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.394+0000 7f1ce6ffd640 1 --2- 172.21.15.16:0/2104367750 >> v2:172.21.15.156:3300/0 conn(0x7f1ce80b2bb0 0x7f1ce80cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.395+0000 7f1ce77fe640 1 --2- 172.21.15.16:0/2104367750 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ce815e020 0x7f1ce80cf270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1cd80097a0 tx=0x7f1cd802a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c24d47a7b604c4b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.395+0000 7f1ce67fc640 1 -- 172.21.15.16:0/2104367750 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8037070 con 0x7f1ce815e020 2024-04-24T23:49:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.395+0000 7f1ce67fc640 1 -- 172.21.15.16:0/2104367750 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cd802ae50 con 0x7f1ce815e020 2024-04-24T23:49:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.395+0000 7f1ce67fc640 1 -- 172.21.15.16:0/2104367750 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8032660 con 0x7f1ce815e020 2024-04-24T23:49:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.396+0000 7f1cedf89640 1 -- 172.21.15.16:0/2104367750 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ce815e020 msgr2=0x7f1ce80cf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.396+0000 7f1cedf89640 1 --2- 172.21.15.16:0/2104367750 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ce815e020 0x7f1ce80cf270 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1cd80097a0 tx=0x7f1cd802a100 comp rx=0 tx=0).stop 2024-04-24T23:49:31.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.396+0000 7f1cedf89640 1 -- 172.21.15.16:0/2104367750 shutdown_connections 2024-04-24T23:49:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.396+0000 7f1cedf89640 1 --2- 172.21.15.16:0/2104367750 >> v2:172.21.15.156:3300/0 conn(0x7f1ce80b2bb0 0x7f1ce80cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.396+0000 7f1cedf89640 1 --2- 172.21.15.16:0/2104367750 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ce815e020 0x7f1ce80cf270 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.396+0000 7f1cedf89640 1 -- 172.21.15.16:0/2104367750 >> 172.21.15.16:0/2104367750 conn(0x7f1ce80d3e10 msgr2=0x7f1ce80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.396+0000 7f1cedf89640 1 -- 172.21.15.16:0/2104367750 shutdown_connections 2024-04-24T23:49:31.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.396+0000 7f1cedf89640 1 -- 172.21.15.16:0/2104367750 wait complete. 2024-04-24T23:49:31.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.397+0000 7f1cedf89640 1 Processor -- start 2024-04-24T23:49:31.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.398+0000 7f1cedf89640 1 -- start start 2024-04-24T23:49:31.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.398+0000 7f1cedf89640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ce80b2bb0 0x7f1ce815b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:31.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.398+0000 7f1cedf89640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ce80cfcf0 0x7f1ce80c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.398+0000 7f1cedf89640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ce815e020 0x7f1ce80b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.398+0000 7f1cedf89640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce815e400 con 0x7f1ce80b2bb0 2024-04-24T23:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.398+0000 7f1cedf89640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce80d1a80 con 0x7f1ce80cfcf0 2024-04-24T23:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.398+0000 7f1cedf89640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1ce80d2fb0 con 0x7f1ce815e020 2024-04-24T23:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.398+0000 7f1ce77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ce80b2bb0 0x7f1ce815b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.398+0000 7f1ce77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ce80b2bb0 0x7f1ce815b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49400/0 (socket says 172.21.15.16:49400) 2024-04-24T23:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.398+0000 7f1ce77fe640 1 -- 172.21.15.16:0/1731564938 learned_addr learned my addr 172.21.15.16:0/1731564938 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1ce6ffd640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ce80cfcf0 0x7f1ce80c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1ce7fff640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ce815e020 0x7f1ce80b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1ce77fe640 1 -- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ce815e020 msgr2=0x7f1ce80b3b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1ce77fe640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ce815e020 0x7f1ce80b3b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1ce77fe640 1 -- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ce80cfcf0 msgr2=0x7f1ce80c7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1ce77fe640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ce80cfcf0 0x7f1ce80c7940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:31.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1ce77fe640 1 -- 172.21.15.16:0/1731564938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce815be00 con 0x7f1ce80b2bb0 2024-04-24T23:49:31.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1ce6ffd640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ce80cfcf0 0x7f1ce80c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:31.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1ce7fff640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ce815e020 0x7f1ce80b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:31.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1ce77fe640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ce80b2bb0 0x7f1ce815b8c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1cd80098c0 tx=0x7f1cd80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:31.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1ccffff640 1 -- 172.21.15.16:0/1731564938 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8041030 con 0x7f1ce80b2bb0 2024-04-24T23:49:31.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1ccffff640 1 -- 172.21.15.16:0/1731564938 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1cd802ac50 con 0x7f1ce80b2bb0 2024-04-24T23:49:31.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1ccffff640 1 -- 172.21.15.16:0/1731564938 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd8037040 con 0x7f1ce80b2bb0 2024-04-24T23:49:31.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1cedf89640 1 -- 172.21.15.16:0/1731564938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ce80c8400 con 0x7f1ce80b2bb0 2024-04-24T23:49:31.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.399+0000 7f1cedf89640 1 -- 172.21.15.16:0/1731564938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ce80c89c0 con 0x7f1ce80b2bb0 2024-04-24T23:49:31.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.400+0000 7f1cedf89640 1 -- 172.21.15.16:0/1731564938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ce806dfe0 con 0x7f1ce80b2bb0 2024-04-24T23:49:31.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.401+0000 7f1ccffff640 1 -- 172.21.15.16:0/1731564938 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1cd8044070 con 0x7f1ce80b2bb0 2024-04-24T23:49:31.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.401+0000 7f1ccffff640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1cd406be30 0x7f1cd406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:31.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.401+0000 7f1ccffff640 1 -- 172.21.15.16:0/1731564938 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f1cd80ab990 con 0x7f1ce80b2bb0 2024-04-24T23:49:31.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.401+0000 7f1ce6ffd640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1cd406be30 0x7f1cd406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:31.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.402+0000 7f1ce6ffd640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1cd406be30 0x7f1cd406e2f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80cda10 tx=0x7f1cdc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:31.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.405+0000 7f1ccffff640 1 -- 172.21.15.16:0/1731564938 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1cd8044e40 con 0x7f1ce80b2bb0 2024-04-24T23:49:31.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.629+0000 7f1cedf89640 1 -- 172.21.15.16:0/1731564938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1ce8159850 con 0x7f1ce80b2bb0 2024-04-24T23:49:31.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.629+0000 7f1ccffff640 1 -- 172.21.15.16:0/1731564938 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f1cd807a090 con 0x7f1ce80b2bb0 2024-04-24T23:49:31.630 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-24T23:49:31.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.631+0000 7f1cedf89640 1 -- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1cd406be30 msgr2=0x7f1cd406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:31.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.631+0000 7f1cedf89640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1cd406be30 0x7f1cd406e2f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80cda10 tx=0x7f1cdc008040 comp rx=0 tx=0).stop 2024-04-24T23:49:31.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.631+0000 7f1cedf89640 1 -- 172.21.15.16:0/1731564938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ce80b2bb0 msgr2=0x7f1ce815b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:31.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.631+0000 7f1cedf89640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ce80b2bb0 0x7f1ce815b8c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1cd80098c0 tx=0x7f1cd80046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:31.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.631+0000 7f1cedf89640 1 -- 172.21.15.16:0/1731564938 shutdown_connections 2024-04-24T23:49:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.631+0000 7f1cedf89640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1cd406be30 0x7f1cd406e2f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.631+0000 7f1cedf89640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1ce815e020 0x7f1ce80b3b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.631+0000 7f1cedf89640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ce80cfcf0 0x7f1ce80c7940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.631+0000 7f1cedf89640 1 --2- 172.21.15.16:0/1731564938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ce80b2bb0 0x7f1ce815b8c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.631+0000 7f1cedf89640 1 -- 172.21.15.16:0/1731564938 >> 172.21.15.16:0/1731564938 conn(0x7f1ce80d3e10 msgr2=0x7f1ce80d2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.632+0000 7f1cedf89640 1 -- 172.21.15.16:0/1731564938 shutdown_connections 2024-04-24T23:49:31.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31.632+0000 7f1cedf89640 1 -- 172.21.15.16:0/1731564938 wait complete. 2024-04-24T23:49:31.642 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T23:49:31.642 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T23:49:31.642 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-04-24T23:49:31.642 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T23:49:31.642 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T23:49:31.642 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T23:49:31.642 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T23:49:31.642 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T23:49:31.642 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T23:49:31.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:31.643 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-04-24T23:49:31.658 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T23:49:31.659 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T23:49:31.659 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T23:49:31.659 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:31.659 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-04-24T23:49:31.659 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:31.659 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:31.660 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:31.660 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-04-24T23:49:31.660 INFO:teuthology.orchestra.run.smithi016.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T23:49:31.660 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:31.660 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-04-24T23:49:31.660 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-24T23:49:31.660 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42704sec preferred_lft 42704sec 2024-04-24T23:49:31.660 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-04-24T23:49:31.660 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:31.661 INFO:teuthology.orchestra.run.smithi016.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:31.661 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:31.661 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-04-24T23:49:31.661 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:31.661 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:31.661 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-04-24T23:49:31.661 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-24T23:49:31.662 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:31.662 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:49:31.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add name ceph-brx type bridge 2024-04-24T23:49:31.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr flush dev ceph-brx 2024-04-24T23:49:31.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set ceph-brx up 2024-04-24T23:49:31.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-24T23:49:31.662 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:49:31.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:31.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:31.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:31.797 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T23:49:31.820 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-04-24T23:49:31.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:31.821 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-04-24T23:49:31.878 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-04-24T23:49:31.878 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T23:49:31.878 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-24T23:49:31.878 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-24T23:49:31.878 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T23:49:31.878 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:31.878 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:49:31.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-24T23:49:31.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-24T23:49:31.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-24T23:49:31.879 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:49:31.968 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:32.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:32.006 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-24T23:49:32.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:32.022 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-04-24T23:49:32.078 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:32.078 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:49:32.078 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:49:32.078 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-24T23:49:32.078 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:49:32.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:32.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:32.195 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-24T23:49:32.195 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:32.195 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:49:32.195 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-24T23:49:32.195 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-24T23:49:32.195 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-24T23:49:32.195 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-24T23:49:32.195 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-24T23:49:32.195 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:49:32.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:32.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:32.307 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:32.307 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:49:32.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-04-24T23:49:32.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.0 master ceph-brx 2024-04-24T23:49:32.307 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:49:32.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:32 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:32.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:32.384 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T23:49:32.384 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:49:32.384 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T23:49:32.409 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T23:49:32.409 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:49:32.409 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T23:49:32.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-24T23:49:32.529 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:49:32.586 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-24T23:49:32.586 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-24T23:49:32.586 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-24T23:49:32.586 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-24T23:49:32.586 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-24T23:49:32.586 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-24T23:49:32.586 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-24T23:49:32.586 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-24T23:49:32.586 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:49:32.587 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:49:32.588 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-24T23:49:32.588 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-24T23:49:32.588 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T23:49:32.588 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T23:49:32.588 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T23:49:32.588 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T23:49:32.588 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-24T23:49:32.588 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:49:32.588 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:49:32.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:32.588 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-24T23:49:32.641 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T23:49:32.641 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T23:49:32.684 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-24T23:49:32.708 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:49:32.767 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-24T23:49:32.767 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-24T23:49:32.767 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-24T23:49:32.767 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-24T23:49:32.767 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-24T23:49:32.767 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-24T23:49:32.767 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:49:32.768 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:49:32.769 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:49:32.769 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:49:32.769 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-24T23:49:32.769 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-24T23:49:32.769 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T23:49:32.769 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T23:49:32.769 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T23:49:32.769 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T23:49:32.769 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-24T23:49:32.769 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:49:32.769 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:49:32.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:32.770 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-24T23:49:32.770 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-04-24T23:49:32.765+0000 7f425c210480 -1 init, newargv = 0x5601330eed50 newargc=15 2024-04-24T23:49:32.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[44700]: starting ceph client 2024-04-24T23:49:32.771 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[44700]: starting fuse 2024-04-24T23:49:32.785 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-04-24T23:49:32.786 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-24T23:49:32.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-04-24T23:49:32.786 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-04-24T23:49:32.786 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-04-24T23:49:32.786 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-04-24T23:49:32.787 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-04-24T23:49:32.787 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:49:32.787 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-04-24T23:49:32.787 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T23:49:32.787 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-04-24T23:49:32.787 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T23:49:32.787 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:49:32.787 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-04-24T23:49:32.787 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-04-24T23:49:32.787 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-04-24T23:49:32.787 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-04-24T23:49:32.787 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:49:32.788 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-04-24T23:49:32.788 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T23:49:32.788 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T23:49:32.788 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T23:49:32.788 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-04-24T23:49:32.788 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-04-24T23:49:32.788 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-04-24T23:49:32.788 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T23:49:32.788 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:49:32.788 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-04-24T23:49:32.788 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-04-24T23:49:32.906 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.44700.asok 2024-04-24T23:49:32.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:32.915 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.44700.asok 2024-04-24T23:49:32.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:32.916 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.44700.asok status 2024-04-24T23:49:33.051 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:49:33.052 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-04-24T23:49:33.052 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-24T23:49:33.052 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-24T23:49:33.052 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-04-24T23:49:33.052 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-04-24T23:49:33.052 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T23:49:33.052 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "44700", 2024-04-24T23:49:33.052 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-04-24T23:49:33.052 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:49:33.052 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-04-24T23:49:33.052 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-04-24T23:49:33.052 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4836, 2024-04-24T23:49:33.052 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-04-24T23:49:33.053 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-04-24T23:49:33.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-04-24T23:49:33.053 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 4836 2024-04-24T23:49:33.053 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:49:33.053 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-04-24T23:49:33.053 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-24T23:49:33.053 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-04-24T23:49:33.054 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 424993722 2024-04-24T23:49:33.054 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:49:33.054 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:49:33.054 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-04-24T23:49:33.055 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-24T23:49:33.055 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-04-24T23:49:33.055 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 424993722 2024-04-24T23:49:33.055 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:49:33.055 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.4836 192.168.0.1:0/424993722", 2024-04-24T23:49:33.055 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/424993722", 2024-04-24T23:49:33.055 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-04-24T23:49:33.055 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 8, 2024-04-24T23:49:33.055 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 25, 2024-04-24T23:49:33.055 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-04-24T23:49:33.055 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-04-24T23:49:33.055 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-04-24T23:49:33.056 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:49:33.060 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T23:49:33.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.161+0000 7f009ade7640 1 Processor -- start 2024-04-24T23:49:33.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.161+0000 7f009ade7640 1 -- start start 2024-04-24T23:49:33.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.162+0000 7f009ade7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f009415dfa0 0x7f00940d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:33.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.162+0000 7f009ade7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00940ce5f0 0x7f00940cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:33.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.162+0000 7f009ade7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00940d05a0 con 0x7f00940b5f00 2024-04-24T23:49:33.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.162+0000 7f009ade7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f00940b7360 con 0x7f009415dfa0 2024-04-24T23:49:33.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.162+0000 7f009ade7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f00940cc640 con 0x7f00940ce5f0 2024-04-24T23:49:33.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.162+0000 7f0098b83640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f009415dfa0 0x7f00940d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:33.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.162+0000 7f0093fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00940ce5f0 0x7f00940cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:33.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.162+0000 7f0098b83640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f009415dfa0 0x7f00940d0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:40660/0 (socket says 172.21.15.16:40660) 2024-04-24T23:49:33.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.162+0000 7f0098b83640 1 -- 172.21.15.16:0/3957494410 learned_addr learned my addr 172.21.15.16:0/3957494410 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:33.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.162+0000 7f0098b83640 1 -- 172.21.15.16:0/3957494410 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00940ce5f0 msgr2=0x7f00940cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:33.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.162+0000 7f0098b83640 1 --2- 172.21.15.16:0/3957494410 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00940ce5f0 0x7f00940cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:33.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.162+0000 7f0098b83640 1 -- 172.21.15.16:0/3957494410 >> v1:172.21.15.156:6789/0 conn(0x7f00940b5f00 legacy=0x7f00940b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:33.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.162+0000 7f0098b83640 1 -- 172.21.15.16:0/3957494410 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00940d0880 con 0x7f009415dfa0 2024-04-24T23:49:33.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.162+0000 7f0093fff640 1 --2- 172.21.15.16:0/3957494410 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00940ce5f0 0x7f00940cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:33.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.163+0000 7f0098b83640 1 --2- 172.21.15.16:0/3957494410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f009415dfa0 0x7f00940d0fe0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0088009870 tx=0x7f008802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0e21cba36122dfb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.163+0000 7f00937fe640 1 -- 172.21.15.16:0/3957494410 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0088037070 con 0x7f009415dfa0 2024-04-24T23:49:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.163+0000 7f00937fe640 1 -- 172.21.15.16:0/3957494410 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0088032440 con 0x7f009415dfa0 2024-04-24T23:49:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.163+0000 7f00937fe640 1 -- 172.21.15.16:0/3957494410 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0088032810 con 0x7f009415dfa0 2024-04-24T23:49:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.164+0000 7f009ade7640 1 -- 172.21.15.16:0/3957494410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f009415dfa0 msgr2=0x7f00940d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.164+0000 7f009ade7640 1 --2- 172.21.15.16:0/3957494410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f009415dfa0 0x7f00940d0fe0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0088009870 tx=0x7f008802a270 comp rx=0 tx=0).stop 2024-04-24T23:49:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.164+0000 7f009ade7640 1 -- 172.21.15.16:0/3957494410 shutdown_connections 2024-04-24T23:49:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.164+0000 7f009ade7640 1 --2- 172.21.15.16:0/3957494410 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00940ce5f0 0x7f00940cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.164+0000 7f009ade7640 1 --2- 172.21.15.16:0/3957494410 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f009415dfa0 0x7f00940d0fe0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.164+0000 7f009ade7640 1 -- 172.21.15.16:0/3957494410 >> 172.21.15.16:0/3957494410 conn(0x7f009406b4d0 msgr2=0x7f00940b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.164+0000 7f009ade7640 1 -- 172.21.15.16:0/3957494410 shutdown_connections 2024-04-24T23:49:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.164+0000 7f009ade7640 1 -- 172.21.15.16:0/3957494410 wait complete. 2024-04-24T23:49:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.165+0000 7f009ade7640 1 Processor -- start 2024-04-24T23:49:33.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.165+0000 7f009ade7640 1 -- start start 2024-04-24T23:49:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.166+0000 7f009ade7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00940b5f00 0x7f00940af350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.166+0000 7f009ade7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00940ce5f0 0x7f009414aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.166+0000 7f009ade7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f009415dfa0 0x7f0094160a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.166+0000 7f009ade7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f009415e380 con 0x7f009415dfa0 2024-04-24T23:49:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.166+0000 7f009ade7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f00940cc2d0 con 0x7f00940b5f00 2024-04-24T23:49:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.166+0000 7f009ade7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f00940d1520 con 0x7f00940ce5f0 2024-04-24T23:49:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.166+0000 7f0099384640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f009415dfa0 0x7f0094160a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.166+0000 7f0099384640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f009415dfa0 0x7f0094160a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49406/0 (socket says 172.21.15.16:49406) 2024-04-24T23:49:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.166+0000 7f0099384640 1 -- 172.21.15.16:0/3143606131 learned_addr learned my addr 172.21.15.16:0/3143606131 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.166+0000 7f0098b83640 1 --2- 172.21.15.16:0/3143606131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00940b5f00 0x7f00940af350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:33.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.166+0000 7f0093fff640 1 --2- 172.21.15.16:0/3143606131 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00940ce5f0 0x7f009414aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.167+0000 7f0099384640 1 -- 172.21.15.16:0/3143606131 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00940ce5f0 msgr2=0x7f009414aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.167+0000 7f0099384640 1 --2- 172.21.15.16:0/3143606131 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00940ce5f0 0x7f009414aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.167+0000 7f0099384640 1 -- 172.21.15.16:0/3143606131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00940b5f00 msgr2=0x7f00940af350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.167+0000 7f0099384640 1 --2- 172.21.15.16:0/3143606131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00940b5f00 0x7f00940af350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.167+0000 7f0099384640 1 -- 172.21.15.16:0/3143606131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00940c66d0 con 0x7f009415dfa0 2024-04-24T23:49:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.167+0000 7f0099384640 1 --2- 172.21.15.16:0/3143606131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f009415dfa0 0x7f0094160a20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f008c00c7d0 tx=0x7f008c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.167+0000 7f00917fa640 1 -- 172.21.15.16:0/3143606131 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c003b70 con 0x7f009415dfa0 2024-04-24T23:49:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.167+0000 7f00917fa640 1 -- 172.21.15.16:0/3143606131 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f008c003d30 con 0x7f009415dfa0 2024-04-24T23:49:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.167+0000 7f00917fa640 1 -- 172.21.15.16:0/3143606131 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f008c00aa90 con 0x7f009415dfa0 2024-04-24T23:49:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.167+0000 7f009ade7640 1 -- 172.21.15.16:0/3143606131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00940c6970 con 0x7f009415dfa0 2024-04-24T23:49:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.167+0000 7f009ade7640 1 -- 172.21.15.16:0/3143606131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00940c7210 con 0x7f009415dfa0 2024-04-24T23:49:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.168+0000 7f009ade7640 1 -- 172.21.15.16:0/3143606131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00940b7890 con 0x7f009415dfa0 2024-04-24T23:49:33.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.168+0000 7f00917fa640 1 -- 172.21.15.16:0/3143606131 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f008c00ac30 con 0x7f009415dfa0 2024-04-24T23:49:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.169+0000 7f00917fa640 1 --2- 172.21.15.16:0/3143606131 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f007806be30 0x7f007806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.169+0000 7f00917fa640 1 -- 172.21.15.16:0/3143606131 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f008c08af90 con 0x7f009415dfa0 2024-04-24T23:49:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.169+0000 7f0098b83640 1 --2- 172.21.15.16:0/3143606131 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f007806be30 0x7f007806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:33.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.169+0000 7f0098b83640 1 --2- 172.21.15.16:0/3143606131 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f007806be30 0x7f007806e2f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0088034000 tx=0x7f008802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:33.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.173+0000 7f00917fa640 1 -- 172.21.15.16:0/3143606131 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f00940b7890 con 0x7f009415dfa0 2024-04-24T23:49:33.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.394+0000 7f009ade7640 1 -- 172.21.15.16:0/3143606131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f009406bea0 con 0x7f009415dfa0 2024-04-24T23:49:33.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.395+0000 7f00917fa640 1 -- 172.21.15.16:0/3143606131 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f008c058fb0 con 0x7f009415dfa0 2024-04-24T23:49:33.395 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-24T23:49:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.397+0000 7f009ade7640 1 -- 172.21.15.16:0/3143606131 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f007806be30 msgr2=0x7f007806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.397+0000 7f009ade7640 1 --2- 172.21.15.16:0/3143606131 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f007806be30 0x7f007806e2f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0088034000 tx=0x7f008802e000 comp rx=0 tx=0).stop 2024-04-24T23:49:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.397+0000 7f009ade7640 1 -- 172.21.15.16:0/3143606131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f009415dfa0 msgr2=0x7f0094160a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.397+0000 7f009ade7640 1 --2- 172.21.15.16:0/3143606131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f009415dfa0 0x7f0094160a20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f008c00c7d0 tx=0x7f008c00cc90 comp rx=0 tx=0).stop 2024-04-24T23:49:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.397+0000 7f009ade7640 1 -- 172.21.15.16:0/3143606131 shutdown_connections 2024-04-24T23:49:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.397+0000 7f009ade7640 1 --2- 172.21.15.16:0/3143606131 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f007806be30 0x7f007806e2f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.397+0000 7f009ade7640 1 --2- 172.21.15.16:0/3143606131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f009415dfa0 0x7f0094160a20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.397+0000 7f009ade7640 1 --2- 172.21.15.16:0/3143606131 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f00940ce5f0 0x7f009414aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.397+0000 7f009ade7640 1 --2- 172.21.15.16:0/3143606131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00940b5f00 0x7f00940af350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.397+0000 7f009ade7640 1 -- 172.21.15.16:0/3143606131 >> 172.21.15.16:0/3143606131 conn(0x7f009406b4d0 msgr2=0x7f00940b4910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.397+0000 7f009ade7640 1 -- 172.21.15.16:0/3143606131 shutdown_connections 2024-04-24T23:49:33.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33.397+0000 7f009ade7640 1 -- 172.21.15.16:0/3143606131 wait complete. 2024-04-24T23:49:33.408 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T23:49:33.408 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T23:49:33.408 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-04-24T23:49:33.408 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-04-24T23:49:33.408 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-04-24T23:49:33.409 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T23:49:33.409 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T23:49:33.409 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T23:49:33.409 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-04-24T23:49:33.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:33.409 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-04-24T23:49:33.425 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T23:49:33.425 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T23:49:33.425 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T23:49:33.425 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:33.425 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-04-24T23:49:33.425 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:33.425 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:33.425 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:33.425 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-04-24T23:49:33.425 INFO:teuthology.orchestra.run.smithi016.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T23:49:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-04-24T23:49:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-24T23:49:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42703sec preferred_lft 42703sec 2024-04-24T23:49:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-04-24T23:49:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:33.426 INFO:teuthology.orchestra.run.smithi016.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-04-24T23:49:33.426 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:33.426 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-04-24T23:49:33.426 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T23:49:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 82:fd:3d:cf:63:61 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T23:49:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::303d:34ff:fea7:344f/64 scope link tentative 2024-04-24T23:49:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:33.427 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T23:49:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 82:fd:3d:cf:63:61 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:49:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::80fd:3dff:fecf:6361/64 scope link 2024-04-24T23:49:33.427 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:33.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:33.428 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T23:49:33.488 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-04-24T23:49:33.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:33.490 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-04-24T23:49:33.545 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-04-24T23:49:33.545 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T23:49:33.545 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-24T23:49:33.546 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-24T23:49:33.546 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T23:49:33.546 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:33.546 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:49:33.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-24T23:49:33.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-24T23:49:33.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-24T23:49:33.546 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:49:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:33.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:33.671 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:33.671 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-24T23:49:33.686 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T23:49:33.686 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:33.687 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list-id 2024-04-24T23:49:33.741 INFO:teuthology.orchestra.run.smithi016.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-04-24T23:49:33.742 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:33.742 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:49:33.742 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T23:49:33.742 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-04-24T23:49:33.742 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:49:33.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:33.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:33.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:33.858 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-24T23:49:33.882 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T23:49:33.882 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T23:49:33.882 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T23:49:33.882 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:33.882 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-04-24T23:49:33.883 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:33.883 INFO:teuthology.orchestra.run.smithi016.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T23:49:33.883 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 9a:f9:b1:cf:6e:63 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-24T23:49:33.883 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-24T23:49:33.883 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:33.883 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::98f9:b1ff:fecf:6e63/64 scope link 2024-04-24T23:49:33.883 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:33.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:33.884 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-04-24T23:49:33.948 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T23:49:33.948 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T23:49:33.948 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T23:49:33.948 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:33.949 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-04-24T23:49:33.949 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:33.949 INFO:teuthology.orchestra.run.smithi016.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T23:49:33.949 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 9a:f9:b1:cf:6e:63 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-24T23:49:33.949 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-24T23:49:33.949 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:33.949 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::98f9:b1ff:fecf:6e63/64 scope link 2024-04-24T23:49:33.949 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:33.950 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-04-24T23:49:33.950 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:33.950 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:49:33.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-04-24T23:49:33.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-24T23:49:33.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-04-24T23:49:33.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-04-24T23:49:33.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-04-24T23:49:33.951 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:49:34.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:34.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:34.100 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:34.101 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:49:34.101 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.1 up 2024-04-24T23:49:34.101 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set dev brx.1 master ceph-brx 2024-04-24T23:49:34.101 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:49:34.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:34.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:34.177 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-04-24T23:49:34.177 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:49:34.177 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-04-24T23:49:34.200 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-04-24T23:49:34.200 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:49:34.200 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-04-24T23:49:34.255 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-24T23:49:34.336 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:49:34.393 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-24T23:49:34.393 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-24T23:49:34.393 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-24T23:49:34.393 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-24T23:49:34.393 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-24T23:49:34.393 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-24T23:49:34.393 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-24T23:49:34.393 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-24T23:49:34.393 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-24T23:49:34.393 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-24T23:49:34.394 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-24T23:49:34.394 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-24T23:49:34.394 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:49:34.395 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-24T23:49:34.395 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-24T23:49:34.395 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-24T23:49:34.395 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-24T23:49:34.395 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-24T23:49:34.395 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-04-24T23:49:34.395 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-24T23:49:34.396 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:49:34.396 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:49:34.396 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:49:34.396 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:49:34.396 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-24T23:49:34.396 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-24T23:49:34.396 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T23:49:34.396 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T23:49:34.396 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T23:49:34.396 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T23:49:34.396 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-24T23:49:34.396 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:49:34.396 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:49:34.397 INFO:teuthology.orchestra.run.smithi016.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-24T23:49:34.397 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T23:49:34.397 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T23:49:34.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:34.398 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-24T23:49:34.449 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-04-24T23:49:34.449 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [43] 2024-04-24T23:49:34.449 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-04-24T23:49:34.492 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-24T23:49:34.517 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:49:34.573 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-24T23:49:34.573 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-24T23:49:34.573 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-24T23:49:34.573 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-24T23:49:34.573 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-24T23:49:34.573 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-24T23:49:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-24T23:49:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-24T23:49:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-24T23:49:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-24T23:49:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-24T23:49:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-24T23:49:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:49:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-24T23:49:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-24T23:49:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-24T23:49:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-24T23:49:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-24T23:49:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:49:34.575 INFO:teuthology.orchestra.run.smithi016.stdout:/home/ubuntu/cephtest/mnt.0 2024-04-24T23:49:34.576 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T23:49:34.576 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T23:49:34.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:34.576 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-24T23:49:34.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi016.stderr:2024-04-24T23:49:34.575+0000 7f7a6ca0d480 -1 init, newargv = 0x559d9a9d9700 newargc=15 2024-04-24T23:49:34.577 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi016.stderr:ceph-fuse[45199]: starting ceph client 2024-04-24T23:49:34.581 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi016.stderr:ceph-fuse[45199]: starting fuse 2024-04-24T23:49:34.593 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-04-24T23:49:34.593 INFO:teuthology.orchestra.run.smithi016.stdout:44 2024-04-24T23:49:34.593 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43, 44] 2024-04-24T23:49:34.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-04-24T23:49:34.594 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-04-24T23:49:34.594 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-04-24T23:49:34.594 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-04-24T23:49:34.594 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-04-24T23:49:34.594 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:49:34.594 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-04-24T23:49:34.594 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-04-24T23:49:34.594 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-04-24T23:49:34.595 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T23:49:34.596 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:49:34.596 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.2")) 2024-04-24T23:49:34.596 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-04-24T23:49:34.713 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.2.45199.asok 2024-04-24T23:49:34.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:34.723 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.45199.asok 2024-04-24T23:49:34.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:34.723 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.45199.asok status 2024-04-24T23:49:34.860 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:49:34.860 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-04-24T23:49:34.860 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-24T23:49:34.860 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-24T23:49:34.860 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "2", 2024-04-24T23:49:34.860 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-04-24T23:49:34.860 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-04-24T23:49:34.860 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "45199", 2024-04-24T23:49:34.860 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-04-24T23:49:34.860 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:49:34.860 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-04-24T23:49:34.860 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4845, 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 4845 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.2:0", 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4107897545 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-24T23:49:34.861 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.2:0", 2024-04-24T23:49:34.862 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 4107897545 2024-04-24T23:49:34.862 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:49:34.862 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.4845 192.168.0.2:0/4107897545", 2024-04-24T23:49:34.862 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.2:0/4107897545", 2024-04-24T23:49:34.862 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-04-24T23:49:34.862 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 8, 2024-04-24T23:49:34.862 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 25, 2024-04-24T23:49:34.862 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-04-24T23:49:34.862 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-04-24T23:49:34.862 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-04-24T23:49:34.862 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:49:34.868 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T23:49:34.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.969+0000 7f445d837640 1 Processor -- start 2024-04-24T23:49:34.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.969+0000 7f445d837640 1 -- start start 2024-04-24T23:49:34.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f445d837640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f44580b2bb0 0x7f44580cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:34.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f445d837640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44580b3390 con 0x7f445815e020 2024-04-24T23:49:34.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f445d837640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f445806bfc0 con 0x7f44580d1fa0 2024-04-24T23:49:34.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f445d837640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f445806c2a0 con 0x7f44580b2bb0 2024-04-24T23:49:34.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f4456ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f445815e020 0x7f44580cf270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38552/0 (socket says 172.21.15.16:38552) 2024-04-24T23:49:34.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f4456ffd640 1 -- 172.21.15.16:0/1408247723 learned_addr learned my addr 172.21.15.16:0/1408247723 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:34.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f44567fc640 1 --2- 172.21.15.16:0/1408247723 >> v2:172.21.15.156:3300/0 conn(0x7f44580b2bb0 0x7f44580cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:34.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f4455ffb640 1 -- 172.21.15.16:0/1408247723 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2860612213 0 0) 0x7f44580b3390 con 0x7f445815e020 2024-04-24T23:49:34.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f4455ffb640 1 -- 172.21.15.16:0/1408247723 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4444003520 con 0x7f445815e020 2024-04-24T23:49:34.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f4455ffb640 1 -- 172.21.15.16:0/1408247723 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3130405575 0 0) 0x7f4444003520 con 0x7f445815e020 2024-04-24T23:49:34.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f4455ffb640 1 -- 172.21.15.16:0/1408247723 >> v2:172.21.15.156:3300/0 conn(0x7f44580b2bb0 msgr2=0x7f44580cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:34.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f4455ffb640 1 --2- 172.21.15.16:0/1408247723 >> v2:172.21.15.156:3300/0 conn(0x7f44580b2bb0 0x7f44580cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:34.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f4455ffb640 1 -- 172.21.15.16:0/1408247723 >> v1:172.21.15.156:6789/0 conn(0x7f44580d1fa0 legacy=0x7f44580d23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:34.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f4455ffb640 1 -- 172.21.15.16:0/1408247723 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f445806c580 con 0x7f445815e020 2024-04-24T23:49:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.970+0000 7f4455ffb640 1 -- 172.21.15.16:0/1408247723 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f4440001670 con 0x7f445815e020 2024-04-24T23:49:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f4455ffb640 1 -- 172.21.15.16:0/1408247723 >> v1:172.21.15.16:6789/0 conn(0x7f445815e020 legacy=0x7f44580cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f4455ffb640 1 --2- 172.21.15.16:0/1408247723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44440039b0 0x7f4444003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f4455ffb640 1 --2- 172.21.15.16:0/1408247723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4444004800 0x7f4444004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f4455ffb640 1 --2- 172.21.15.16:0/1408247723 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44440051c0 0x7f444400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f4455ffb640 1 -- 172.21.15.16:0/1408247723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f445806c2a0 con 0x7f4444004800 2024-04-24T23:49:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f4455ffb640 1 -- 172.21.15.16:0/1408247723 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f444400c160 con 0x7f44440039b0 2024-04-24T23:49:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f4455ffb640 1 -- 172.21.15.16:0/1408247723 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f444400c360 con 0x7f44440051c0 2024-04-24T23:49:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f4456ffd640 1 --2- 172.21.15.16:0/1408247723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44440039b0 0x7f4444003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f44567fc640 1 --2- 172.21.15.16:0/1408247723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4444004800 0x7f4444004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f44577fe640 1 --2- 172.21.15.16:0/1408247723 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44440051c0 0x7f444400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f44567fc640 1 -- 172.21.15.16:0/1408247723 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44440051c0 msgr2=0x7f444400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:34.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f44567fc640 1 --2- 172.21.15.16:0/1408247723 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f44440051c0 0x7f444400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:34.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f44567fc640 1 -- 172.21.15.16:0/1408247723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44440039b0 msgr2=0x7f4444003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:34.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f44567fc640 1 --2- 172.21.15.16:0/1408247723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44440039b0 0x7f4444003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:34.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f44577fe640 1 -- 172.21.15.16:0/1408247723 reap_dead start 2024-04-24T23:49:34.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f44567fc640 1 -- 172.21.15.16:0/1408247723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4444003520 con 0x7f4444004800 2024-04-24T23:49:34.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.971+0000 7f4456ffd640 1 --2- 172.21.15.16:0/1408247723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f44440039b0 0x7f4444003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:34.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.972+0000 7f44567fc640 1 --2- 172.21.15.16:0/1408247723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4444004800 0x7f4444004c80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4448009ac0 tx=0x7f4444005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a79a62e1f33cf470 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:34.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.972+0000 7f4455ffb640 1 -- 172.21.15.16:0/1408247723 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4448036070 con 0x7f4444004800 2024-04-24T23:49:34.975 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.972+0000 7f4455ffb640 1 -- 172.21.15.16:0/1408247723 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4448031400 con 0x7f4444004800 2024-04-24T23:49:34.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.972+0000 7f4455ffb640 1 -- 172.21.15.16:0/1408247723 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4448009690 con 0x7f4444004800 2024-04-24T23:49:34.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.973+0000 7f445d837640 1 -- 172.21.15.16:0/1408247723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4444004800 msgr2=0x7f4444004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:34.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.973+0000 7f445d837640 1 --2- 172.21.15.16:0/1408247723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4444004800 0x7f4444004c80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4448009ac0 tx=0x7f4444005830 comp rx=0 tx=0).stop 2024-04-24T23:49:34.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.973+0000 7f445d837640 1 -- 172.21.15.16:0/1408247723 shutdown_connections 2024-04-24T23:49:34.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.973+0000 7f445d837640 1 --2- 172.21.15.16:0/1408247723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4444004800 0x7f4444004c80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:34.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.973+0000 7f445d837640 1 -- 172.21.15.16:0/1408247723 >> 172.21.15.16:0/1408247723 conn(0x7f44580d3e10 msgr2=0x7f44580d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:34.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.973+0000 7f445d837640 1 -- 172.21.15.16:0/1408247723 shutdown_connections 2024-04-24T23:49:34.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.973+0000 7f445d837640 1 -- 172.21.15.16:0/1408247723 wait complete. 2024-04-24T23:49:34.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.974+0000 7f445d837640 1 Processor -- start 2024-04-24T23:49:34.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.974+0000 7f445d837640 1 -- start start 2024-04-24T23:49:34.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f445d837640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4444004800 0x7f445815eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:34.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f445d837640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f445815f0e0 0x7f4458145a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:34.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f445d837640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4458145f90 0x7f4458146440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:34.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f445d837640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f445815e400 con 0x7f4444004800 2024-04-24T23:49:34.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f445d837640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f44580d1a80 con 0x7f4458145f90 2024-04-24T23:49:34.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f445d837640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f44580cf270 con 0x7f445815f0e0 2024-04-24T23:49:34.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f4456ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4444004800 0x7f445815eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:34.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f4456ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4444004800 0x7f445815eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49424/0 (socket says 172.21.15.16:49424) 2024-04-24T23:49:34.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f4456ffd640 1 -- 172.21.15.16:0/2006189905 learned_addr learned my addr 172.21.15.16:0/2006189905 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:34.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f44567fc640 1 --2- 172.21.15.16:0/2006189905 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f445815f0e0 0x7f4458145a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:34.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f44577fe640 1 --2- 172.21.15.16:0/2006189905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4458145f90 0x7f4458146440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:34.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f4456ffd640 1 -- 172.21.15.16:0/2006189905 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f445815f0e0 msgr2=0x7f4458145a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:34.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f4456ffd640 1 --2- 172.21.15.16:0/2006189905 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f445815f0e0 0x7f4458145a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:34.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f4456ffd640 1 -- 172.21.15.16:0/2006189905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4458145f90 msgr2=0x7f4458146440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:34.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f4456ffd640 1 --2- 172.21.15.16:0/2006189905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4458145f90 0x7f4458146440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:34.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.975+0000 7f4456ffd640 1 -- 172.21.15.16:0/2006189905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4458154110 con 0x7f4444004800 2024-04-24T23:49:34.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.976+0000 7f4456ffd640 1 --2- 172.21.15.16:0/2006189905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4444004800 0x7f445815eba0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4440001200 tx=0x7f4440007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:34.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.976+0000 7f444ffff640 1 -- 172.21.15.16:0/2006189905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4440007cf0 con 0x7f4444004800 2024-04-24T23:49:34.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.976+0000 7f444ffff640 1 -- 172.21.15.16:0/2006189905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4440018070 con 0x7f4444004800 2024-04-24T23:49:34.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.976+0000 7f445d837640 1 -- 172.21.15.16:0/2006189905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f445815f920 con 0x7f4444004800 2024-04-24T23:49:34.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.976+0000 7f444ffff640 1 -- 172.21.15.16:0/2006189905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444000fc20 con 0x7f4444004800 2024-04-24T23:49:34.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.976+0000 7f445d837640 1 -- 172.21.15.16:0/2006189905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f445815fee0 con 0x7f4444004800 2024-04-24T23:49:34.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.977+0000 7f445d837640 1 -- 172.21.15.16:0/2006189905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f445806d960 con 0x7f4444004800 2024-04-24T23:49:34.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.978+0000 7f444ffff640 1 -- 172.21.15.16:0/2006189905 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f444001d460 con 0x7f4444004800 2024-04-24T23:49:34.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.978+0000 7f444ffff640 1 --2- 172.21.15.16:0/2006189905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f443806be30 0x7f443806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:34.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.978+0000 7f444ffff640 1 -- 172.21.15.16:0/2006189905 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6532+0+0 (secure 0 0 0) 0x7f444008afa0 con 0x7f4444004800 2024-04-24T23:49:34.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.978+0000 7f44567fc640 1 --2- 172.21.15.16:0/2006189905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f443806be30 0x7f443806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:34.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.979+0000 7f44567fc640 1 --2- 172.21.15.16:0/2006189905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f443806be30 0x7f443806e2f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4448033000 tx=0x7f4448008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:34.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:34.983+0000 7f444ffff640 1 -- 172.21.15.16:0/2006189905 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f445806d960 con 0x7f4444004800 2024-04-24T23:49:35.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:35.205+0000 7f445d837640 1 -- 172.21.15.16:0/2006189905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f445806dfe0 con 0x7f4444004800 2024-04-24T23:49:35.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:35.206+0000 7f444ffff640 1 -- 172.21.15.16:0/2006189905 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f4440059050 con 0x7f4444004800 2024-04-24T23:49:35.207 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-24T23:49:35.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:35.208+0000 7f445d837640 1 -- 172.21.15.16:0/2006189905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f443806be30 msgr2=0x7f443806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:35.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:35.208+0000 7f445d837640 1 --2- 172.21.15.16:0/2006189905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f443806be30 0x7f443806e2f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4448033000 tx=0x7f4448008030 comp rx=0 tx=0).stop 2024-04-24T23:49:35.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:35.208+0000 7f445d837640 1 -- 172.21.15.16:0/2006189905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4444004800 msgr2=0x7f445815eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:35.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:35.208+0000 7f445d837640 1 --2- 172.21.15.16:0/2006189905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4444004800 0x7f445815eba0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4440001200 tx=0x7f4440007720 comp rx=0 tx=0).stop 2024-04-24T23:49:35.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:35.208+0000 7f445d837640 1 -- 172.21.15.16:0/2006189905 shutdown_connections 2024-04-24T23:49:35.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:35.208+0000 7f445d837640 1 --2- 172.21.15.16:0/2006189905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f443806be30 0x7f443806e2f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:35.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:35.208+0000 7f445d837640 1 --2- 172.21.15.16:0/2006189905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4458145f90 0x7f4458146440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:35.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:35.208+0000 7f445d837640 1 --2- 172.21.15.16:0/2006189905 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f445815f0e0 0x7f4458145a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:35.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:35.208+0000 7f445d837640 1 --2- 172.21.15.16:0/2006189905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4444004800 0x7f445815eba0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:35.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:35.208+0000 7f445d837640 1 -- 172.21.15.16:0/2006189905 >> 172.21.15.16:0/2006189905 conn(0x7f44580d3e10 msgr2=0x7f44580ca310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:35.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:35.208+0000 7f445d837640 1 -- 172.21.15.16:0/2006189905 shutdown_connections 2024-04-24T23:49:35.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:35.208+0000 7f445d837640 1 -- 172.21.15.16:0/2006189905 wait complete. 2024-04-24T23:49:35.223 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T23:49:35.223 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T23:49:35.223 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi156.front.sepia.ceph.com 2024-04-24T23:49:35.223 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-04-24T23:49:35.223 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-04-24T23:49:35.223 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T23:49:35.223 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T23:49:35.223 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T23:49:35.224 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-04-24T23:49:35.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:35.224 DEBUG:teuthology.orchestra.run.smithi156:> ip addr 2024-04-24T23:49:35.243 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T23:49:35.243 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T23:49:35.243 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T23:49:35.243 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:35.243 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2024-04-24T23:49:35.243 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:35.243 INFO:teuthology.orchestra.run.smithi156.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:35.244 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c0 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:35.244 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f0 2024-04-24T23:49:35.244 INFO:teuthology.orchestra.run.smithi156.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:35.244 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c1 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:35.244 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f1 2024-04-24T23:49:35.244 INFO:teuthology.orchestra.run.smithi156.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:35.244 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fc brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:35.244 INFO:teuthology.orchestra.run.smithi156.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T23:49:35.244 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fd brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:35.244 INFO:teuthology.orchestra.run.smithi156.stdout: inet 172.21.15.156/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-24T23:49:35.244 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft 42718sec preferred_lft 42718sec 2024-04-24T23:49:35.244 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::771b:f6fc:23af:4f9c/64 scope link noprefixroute 2024-04-24T23:49:35.244 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:35.245 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-24T23:49:35.245 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:35.245 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-04-24T23:49:35.245 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link add name ceph-brx type bridge 2024-04-24T23:49:35.245 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip addr flush dev ceph-brx 2024-04-24T23:49:35.245 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set ceph-brx up 2024-04-24T23:49:35.245 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-24T23:49:35.245 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-04-24T23:49:35.338 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:35.558 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:35.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:35.565 DEBUG:teuthology.orchestra.run.smithi156:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T23:49:35.589 INFO:teuthology.orchestra.run.smithi156.stdout:1 2024-04-24T23:49:35.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:35.591 DEBUG:teuthology.orchestra.run.smithi156:> route 2024-04-24T23:49:35.666 INFO:teuthology.orchestra.run.smithi156.stdout:Kernel IP routing table 2024-04-24T23:49:35.666 INFO:teuthology.orchestra.run.smithi156.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T23:49:35.666 INFO:teuthology.orchestra.run.smithi156.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T23:49:35.666 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T23:49:35.666 INFO:teuthology.orchestra.run.smithi156.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T23:49:35.667 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:35.667 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-04-24T23:49:35.667 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T23:49:35.667 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T23:49:35.667 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T23:49:35.667 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-04-24T23:49:35.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:35 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:35.927 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:35.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:35.933 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2024-04-24T23:49:35.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:35.948 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list-id 2024-04-24T23:49:36.003 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:36.003 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-04-24T23:49:36.003 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T23:49:36.003 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-04-24T23:49:36.003 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-04-24T23:49:36.095 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:36.119 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:36.125 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-04-24T23:49:36.126 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:36.126 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-04-24T23:49:36.126 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-04-24T23:49:36.126 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-24T23:49:36.126 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-04-24T23:49:36.126 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-04-24T23:49:36.126 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-04-24T23:49:36.126 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-04-24T23:49:36.174 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:36.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:36.244 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:36.244 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-04-24T23:49:36.244 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set brx.0 up 2024-04-24T23:49:36.244 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set dev brx.0 master ceph-brx 2024-04-24T23:49:36.244 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-04-24T23:49:36.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:36 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:36.314 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:36.321 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-04-24T23:49:36.322 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:49:36.322 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-04-24T23:49:36.336 INFO:teuthology.orchestra.run.smithi156.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-04-24T23:49:36.336 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:49:36.336 DEBUG:teuthology.orchestra.run.smithi156:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-04-24T23:49:36.391 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-04-24T23:49:36.459 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:49:36.514 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-04-24T23:49:36.514 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-04-24T23:49:36.514 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-04-24T23:49:36.514 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-04-24T23:49:36.514 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-04-24T23:49:36.515 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-04-24T23:49:36.515 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-04-24T23:49:36.515 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-04-24T23:49:36.515 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-04-24T23:49:36.515 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-04-24T23:49:36.515 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-04-24T23:49:36.515 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-04-24T23:49:36.515 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:49:36.515 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-04-24T23:49:36.515 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-04-24T23:49:36.515 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-04-24T23:49:36.515 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-04-24T23:49:36.515 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysusers.service 2024-04-24T23:49:36.515 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-04-24T23:49:36.516 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:49:36.516 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-04-24T23:49:36.516 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:49:36.516 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:49:36.516 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:49:36.516 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-04-24T23:49:36.516 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-04-24T23:49:36.516 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2024-04-24T23:49:36.516 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2024-04-24T23:49:36.516 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2024-04-24T23:49:36.516 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2024-04-24T23:49:36.516 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-04-24T23:49:36.516 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T23:49:36.516 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T23:49:36.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:36.517 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-04-24T23:49:36.569 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T23:49:36.569 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-04-24T23:49:36.612 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-04-24T23:49:36.639 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:49:36.694 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-04-24T23:49:36.695 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-04-24T23:49:36.695 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-04-24T23:49:36.695 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-04-24T23:49:36.695 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-04-24T23:49:36.695 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-04-24T23:49:36.695 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-04-24T23:49:36.695 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-04-24T23:49:36.695 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-04-24T23:49:36.695 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-04-24T23:49:36.695 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-04-24T23:49:36.695 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysusers.service 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2024-04-24T23:49:36.696 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2024-04-24T23:49:36.697 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2024-04-24T23:49:36.697 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2024-04-24T23:49:36.697 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-04-24T23:49:36.697 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T23:49:36.697 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T23:49:36.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:36.698 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-04-24T23:49:36.701 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:2024-04-24T23:49:36.702+0000 7f100f7ee480 -1 init, newargv = 0x55cfde6dc170 newargc=15 2024-04-24T23:49:36.702 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[40732]: starting ceph client 2024-04-24T23:49:36.707 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi156.stderr:ceph-fuse[40732]: starting fuse 2024-04-24T23:49:36.716 INFO:teuthology.orchestra.run.smithi156.stdout:42 2024-04-24T23:49:36.716 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42] 2024-04-24T23:49:36.717 DEBUG:teuthology.orchestra.run.smithi156:> sudo stdin-killer -- python3 -c ' 2024-04-24T23:49:36.717 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2024-04-24T23:49:36.717 DEBUG:teuthology.orchestra.run.smithi156:> import re 2024-04-24T23:49:36.717 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-04-24T23:49:36.717 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2024-04-24T23:49:36.717 DEBUG:teuthology.orchestra.run.smithi156:> 2024-04-24T23:49:36.717 DEBUG:teuthology.orchestra.run.smithi156:> def _find_admin_socket(client_name): 2024-04-24T23:49:36.717 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-04-24T23:49:36.717 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2024-04-24T23:49:36.717 DEBUG:teuthology.orchestra.run.smithi156:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-04-24T23:49:36.717 DEBUG:teuthology.orchestra.run.smithi156:> 2024-04-24T23:49:36.717 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2024-04-24T23:49:36.717 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2024-04-24T23:49:36.718 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2024-04-24T23:49:36.718 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2024-04-24T23:49:36.718 DEBUG:teuthology.orchestra.run.smithi156:> 2024-04-24T23:49:36.718 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2024-04-24T23:49:36.718 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T23:49:36.718 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T23:49:36.718 DEBUG:teuthology.orchestra.run.smithi156:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T23:49:36.718 DEBUG:teuthology.orchestra.run.smithi156:> contents = proc_f.read() 2024-04-24T23:49:36.718 DEBUG:teuthology.orchestra.run.smithi156:> if mountpoint in contents: 2024-04-24T23:49:36.718 DEBUG:teuthology.orchestra.run.smithi156:> return f 2024-04-24T23:49:36.718 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T23:49:36.718 DEBUG:teuthology.orchestra.run.smithi156:> 2024-04-24T23:49:36.718 DEBUG:teuthology.orchestra.run.smithi156:> print(_find_admin_socket("client.1")) 2024-04-24T23:49:36.718 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-04-24T23:49:36.833 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.1.40732.asok 2024-04-24T23:49:36.836 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:36.844 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.40732.asok 2024-04-24T23:49:36.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:36.844 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.40732.asok status 2024-04-24T23:49:36.979 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-04-24T23:49:36.979 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2024-04-24T23:49:36.979 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-24T23:49:36.979 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-24T23:49:36.979 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "1", 2024-04-24T23:49:36.979 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2024-04-24T23:49:36.979 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-04-24T23:49:36.979 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "40732", 2024-04-24T23:49:36.979 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2024-04-24T23:49:36.979 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-24T23:49:36.979 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2024-04-24T23:49:36.979 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2024-04-24T23:49:36.980 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 4610, 2024-04-24T23:49:36.980 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2024-04-24T23:49:36.980 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2024-04-24T23:49:36.980 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2024-04-24T23:49:36.980 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 4610 2024-04-24T23:49:36.980 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-24T23:49:36.980 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-04-24T23:49:36.980 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-04-24T23:49:36.980 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2024-04-24T23:49:36.980 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 819927961 2024-04-24T23:49:36.980 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-24T23:49:36.980 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-24T23:49:36.980 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-04-24T23:49:36.980 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-04-24T23:49:36.981 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.1:0", 2024-04-24T23:49:36.981 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 819927961 2024-04-24T23:49:36.981 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-24T23:49:36.981 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.4610 192.168.0.1:0/819927961", 2024-04-24T23:49:36.981 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "192.168.0.1:0/819927961", 2024-04-24T23:49:36.981 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2024-04-24T23:49:36.981 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 8, 2024-04-24T23:49:36.981 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 27, 2024-04-24T23:49:36.981 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2024-04-24T23:49:36.981 INFO:teuthology.orchestra.run.smithi156.stdout: "blocklisted": false, 2024-04-24T23:49:36.981 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2024-04-24T23:49:36.981 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-04-24T23:49:36.988 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T23:49:37.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.100+0000 7f82ff1f9640 1 Processor -- start 2024-04-24T23:49:37.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.100+0000 7f82ff1f9640 1 -- start start 2024-04-24T23:49:37.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.101+0000 7f82ff1f9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f82f806e910 0x7f82f8072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:37.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.101+0000 7f82ff1f9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82f80727d0 0x7f82f80ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:37.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.101+0000 7f82ff1f9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82f80b3300 con 0x7f82f815a450 2024-04-24T23:49:37.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.101+0000 7f82ff1f9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f82f8072c40 con 0x7f82f806e910 2024-04-24T23:49:37.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.101+0000 7f82ff1f9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f82f80b35e0 con 0x7f82f80727d0 2024-04-24T23:49:37.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.102+0000 7f82f7fff640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f82f806e910 0x7f82f8072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:37.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.102+0000 7f82fcf95640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f82f815a450 0x7f82f80b2a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:52960/0 (socket says 172.21.15.16:52960) 2024-04-24T23:49:37.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.102+0000 7f82fd796640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82f80727d0 0x7f82f80ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:37.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.102+0000 7f82fcf95640 1 -- 172.21.15.16:0/702629015 learned_addr learned my addr 172.21.15.16:0/702629015 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:37.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.102+0000 7f82f77fe640 1 -- 172.21.15.16:0/702629015 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4013276713 0 0) 0x7f82f80b3300 con 0x7f82f815a450 2024-04-24T23:49:37.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.102+0000 7f82f77fe640 1 -- 172.21.15.16:0/702629015 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f82e0003520 con 0x7f82f815a450 2024-04-24T23:49:37.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.102+0000 7f82f7fff640 1 -- 172.21.15.16:0/702629015 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82f80727d0 msgr2=0x7f82f80ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:37.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.102+0000 7f82f7fff640 1 --2- 172.21.15.16:0/702629015 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82f80727d0 0x7f82f80ccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:37.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.102+0000 7f82f7fff640 1 -- 172.21.15.16:0/702629015 >> v1:172.21.15.156:6789/0 conn(0x7f82f815a450 legacy=0x7f82f80b2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:37.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.102+0000 7f82f7fff640 1 -- 172.21.15.16:0/702629015 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f80b38c0 con 0x7f82f806e910 2024-04-24T23:49:37.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.102+0000 7f82fd796640 1 --2- 172.21.15.16:0/702629015 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82f80727d0 0x7f82f80ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:37.104 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.103+0000 7f82f7fff640 1 --2- 172.21.15.16:0/702629015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82f806e910 0x7f82f8072290 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f82ec00a940 tx=0x7f82ec02b4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abfb0571aa5b64e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:37.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.103+0000 7f82f77fe640 1 -- 172.21.15.16:0/702629015 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec00b040 con 0x7f82f806e910 2024-04-24T23:49:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.103+0000 7f82f77fe640 1 -- 172.21.15.16:0/702629015 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82ec030710 con 0x7f82f806e910 2024-04-24T23:49:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.103+0000 7f82f77fe640 1 -- 172.21.15.16:0/702629015 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec030ac0 con 0x7f82f806e910 2024-04-24T23:49:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.103+0000 7f82ff1f9640 1 -- 172.21.15.16:0/702629015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82f806e910 msgr2=0x7f82f8072290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.103+0000 7f82ff1f9640 1 --2- 172.21.15.16:0/702629015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82f806e910 0x7f82f8072290 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f82ec00a940 tx=0x7f82ec02b4c0 comp rx=0 tx=0).stop 2024-04-24T23:49:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.103+0000 7f82ff1f9640 1 -- 172.21.15.16:0/702629015 shutdown_connections 2024-04-24T23:49:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.103+0000 7f82ff1f9640 1 --2- 172.21.15.16:0/702629015 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82f80727d0 0x7f82f80ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.103+0000 7f82ff1f9640 1 --2- 172.21.15.16:0/702629015 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82f806e910 0x7f82f8072290 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.103+0000 7f82ff1f9640 1 -- 172.21.15.16:0/702629015 >> 172.21.15.16:0/702629015 conn(0x7f82f80b2000 msgr2=0x7f82f80cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.104+0000 7f82ff1f9640 1 -- 172.21.15.16:0/702629015 shutdown_connections 2024-04-24T23:49:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.104+0000 7f82ff1f9640 1 -- 172.21.15.16:0/702629015 wait complete. 2024-04-24T23:49:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.105+0000 7f82ff1f9640 1 Processor -- start 2024-04-24T23:49:37.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.105+0000 7f82ff1f9640 1 -- start start 2024-04-24T23:49:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82ff1f9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82f806e910 0x7f82f8156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82ff1f9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82f80727d0 0x7f82f814c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82ff1f9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f815a450 0x7f82f81f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82ff1f9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f82f8071a00 con 0x7f82f815a450 2024-04-24T23:49:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82ff1f9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f82f806bb60 con 0x7f82f80727d0 2024-04-24T23:49:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82ff1f9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f82f8071b80 con 0x7f82f806e910 2024-04-24T23:49:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82fd796640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f815a450 0x7f82f81f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82fd796640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f815a450 0x7f82f81f24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59594/0 (socket says 172.21.15.16:59594) 2024-04-24T23:49:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82fd796640 1 -- 172.21.15.16:0/2537572194 learned_addr learned my addr 172.21.15.16:0/2537572194 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82f7fff640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82f80727d0 0x7f82f814c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82fcf95640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82f806e910 0x7f82f8156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:37.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82f7fff640 1 -- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82f806e910 msgr2=0x7f82f8156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:37.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82f7fff640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82f806e910 0x7f82f8156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:37.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82f7fff640 1 -- 172.21.15.16:0/2537572194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f815a450 msgr2=0x7f82f81f24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:37.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82f7fff640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f815a450 0x7f82f81f24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:37.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82f7fff640 1 -- 172.21.15.16:0/2537572194 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f8154560 con 0x7f82f80727d0 2024-04-24T23:49:37.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82fd796640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f815a450 0x7f82f81f24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:37.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.106+0000 7f82f7fff640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82f80727d0 0x7f82f814c290 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f82ec032000 tx=0x7f82ec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:37.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.107+0000 7f82f57fa640 1 -- 172.21.15.16:0/2537572194 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec03f070 con 0x7f82f80727d0 2024-04-24T23:49:37.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.107+0000 7f82f57fa640 1 -- 172.21.15.16:0/2537572194 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82ec033a50 con 0x7f82f80727d0 2024-04-24T23:49:37.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.107+0000 7f82f57fa640 1 -- 172.21.15.16:0/2537572194 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82ec033e00 con 0x7f82f80727d0 2024-04-24T23:49:37.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.107+0000 7f82ff1f9640 1 -- 172.21.15.16:0/2537572194 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f814f040 con 0x7f82f80727d0 2024-04-24T23:49:37.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.107+0000 7f82ff1f9640 1 -- 172.21.15.16:0/2537572194 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f8157290 con 0x7f82f80727d0 2024-04-24T23:49:37.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.108+0000 7f82ff1f9640 1 -- 172.21.15.16:0/2537572194 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82f81548e0 con 0x7f82f80727d0 2024-04-24T23:49:37.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.108+0000 7f82f57fa640 1 -- 172.21.15.16:0/2537572194 <== mon.1 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f82ec042070 con 0x7f82f80727d0 2024-04-24T23:49:37.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.109+0000 7f82f57fa640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f82e406be30 0x7f82e406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:37.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.109+0000 7f82f57fa640 1 -- 172.21.15.16:0/2537572194 <== mon.1 v2:172.21.15.156:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f82ec0abe80 con 0x7f82f80727d0 2024-04-24T23:49:37.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.109+0000 7f82fcf95640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f82e406be30 0x7f82e406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:37.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.109+0000 7f82fcf95640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f82e406be30 0x7f82e406e2f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f82f0005c00 tx=0x7f82f0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:37.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.113+0000 7f82f57fa640 1 -- 172.21.15.16:0/2537572194 <== mon.1 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f82ec079dd0 con 0x7f82f80727d0 2024-04-24T23:49:37.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.336+0000 7f82ff1f9640 1 -- 172.21.15.16:0/2537572194 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f82f80b46e0 con 0x7f82f80727d0 2024-04-24T23:49:37.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.336+0000 7f82f57fa640 1 -- 172.21.15.16:0/2537572194 <== mon.1 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) ==== 53+0+88 (secure 0 0 0) 0x7f82f80b46e0 con 0x7f82f80727d0 2024-04-24T23:49:37.337 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-24T23:49:37.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.338+0000 7f82ff1f9640 1 -- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f82e406be30 msgr2=0x7f82e406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:37.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.338+0000 7f82ff1f9640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f82e406be30 0x7f82e406e2f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f82f0005c00 tx=0x7f82f0009040 comp rx=0 tx=0).stop 2024-04-24T23:49:37.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.338+0000 7f82ff1f9640 1 -- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82f80727d0 msgr2=0x7f82f814c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:37.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.338+0000 7f82ff1f9640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82f80727d0 0x7f82f814c290 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f82ec032000 tx=0x7f82ec0046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:37.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.338+0000 7f82ff1f9640 1 -- 172.21.15.16:0/2537572194 shutdown_connections 2024-04-24T23:49:37.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.338+0000 7f82ff1f9640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f82e406be30 0x7f82e406e2f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:37.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.338+0000 7f82ff1f9640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82f815a450 0x7f82f81f24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:37.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.338+0000 7f82ff1f9640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f82f80727d0 0x7f82f814c290 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:37.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.338+0000 7f82ff1f9640 1 --2- 172.21.15.16:0/2537572194 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f82f806e910 0x7f82f8156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:37.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.338+0000 7f82ff1f9640 1 -- 172.21.15.16:0/2537572194 >> 172.21.15.16:0/2537572194 conn(0x7f82f80b2000 msgr2=0x7f82f815dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:37.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.339+0000 7f82ff1f9640 1 -- 172.21.15.16:0/2537572194 shutdown_connections 2024-04-24T23:49:37.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:37.339+0000 7f82ff1f9640 1 -- 172.21.15.16:0/2537572194 wait complete. 2024-04-24T23:49:37.350 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T23:49:37.350 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T23:49:37.350 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi156.front.sepia.ceph.com 2024-04-24T23:49:37.350 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-04-24T23:49:37.350 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-04-24T23:49:37.350 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T23:49:37.350 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T23:49:37.350 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T23:49:37.350 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-04-24T23:49:37.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:37.351 DEBUG:teuthology.orchestra.run.smithi156:> ip addr 2024-04-24T23:49:37.367 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T23:49:37.367 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T23:49:37.367 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T23:49:37.367 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:37.367 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2024-04-24T23:49:37.367 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:37.368 INFO:teuthology.orchestra.run.smithi156.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:37.368 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c0 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:37.368 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f0 2024-04-24T23:49:37.368 INFO:teuthology.orchestra.run.smithi156.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:37.368 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:da:6a:c1 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:37.368 INFO:teuthology.orchestra.run.smithi156.stdout: altname enp7s0f1 2024-04-24T23:49:37.368 INFO:teuthology.orchestra.run.smithi156.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:49:37.368 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fc brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:37.368 INFO:teuthology.orchestra.run.smithi156.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T23:49:37.368 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 0c:c4:7a:88:6d:fd brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:37.368 INFO:teuthology.orchestra.run.smithi156.stdout: inet 172.21.15.156/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-04-24T23:49:37.368 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft 42716sec preferred_lft 42716sec 2024-04-24T23:49:37.368 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::771b:f6fc:23af:4f9c/64 scope link noprefixroute 2024-04-24T23:49:37.368 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:37.369 INFO:teuthology.orchestra.run.smithi156.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T23:49:37.369 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 9e:21:9e:60:f7:66 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:49:37.369 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T23:49:37.369 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:37.369 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::40f0:ffff:fe3c:75a4/64 scope link tentative 2024-04-24T23:49:37.369 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:37.369 INFO:teuthology.orchestra.run.smithi156.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T23:49:37.369 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 9e:21:9e:60:f7:66 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T23:49:37.369 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::9c21:9eff:fe60:f766/64 scope link tentative 2024-04-24T23:49:37.369 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:37.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:37.369 DEBUG:teuthology.orchestra.run.smithi156:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T23:49:37.430 INFO:teuthology.orchestra.run.smithi156.stdout:1 2024-04-24T23:49:37.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:37.431 DEBUG:teuthology.orchestra.run.smithi156:> route 2024-04-24T23:49:37.487 INFO:teuthology.orchestra.run.smithi156.stdout:Kernel IP routing table 2024-04-24T23:49:37.487 INFO:teuthology.orchestra.run.smithi156.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T23:49:37.487 INFO:teuthology.orchestra.run.smithi156.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-04-24T23:49:37.487 INFO:teuthology.orchestra.run.smithi156.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-04-24T23:49:37.487 INFO:teuthology.orchestra.run.smithi156.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T23:49:37.488 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:37.488 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-04-24T23:49:37.488 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-04-24T23:49:37.488 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-04-24T23:49:37.488 DEBUG:teuthology.orchestra.run.smithi156:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-04-24T23:49:37.488 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-04-24T23:49:37.576 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:37.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:37.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:37.612 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2024-04-24T23:49:37.627 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-24T23:49:37.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:37.628 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list-id 2024-04-24T23:49:37.683 INFO:teuthology.orchestra.run.smithi156.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-04-24T23:49:37.683 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:37.683 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-04-24T23:49:37.683 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-24T23:49:37.683 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-04-24T23:49:37.683 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-04-24T23:49:37.772 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:37.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:37 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:37.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:37.799 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-24T23:49:37.822 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T23:49:37.822 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T23:49:37.822 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T23:49:37.822 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:37.822 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2024-04-24T23:49:37.822 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:37.822 INFO:teuthology.orchestra.run.smithi156.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T23:49:37.823 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 5a:69:19:79:bb:9e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-24T23:49:37.823 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-24T23:49:37.823 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:37.823 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::5869:19ff:fe79:bb9e/64 scope link 2024-04-24T23:49:37.823 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:37.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:37.824 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-04-24T23:49:37.888 INFO:teuthology.orchestra.run.smithi156.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T23:49:37.888 INFO:teuthology.orchestra.run.smithi156.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T23:49:37.888 INFO:teuthology.orchestra.run.smithi156.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T23:49:37.889 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:37.889 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 ::1/128 scope host 2024-04-24T23:49:37.889 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:37.889 INFO:teuthology.orchestra.run.smithi156.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T23:49:37.889 INFO:teuthology.orchestra.run.smithi156.stdout: link/ether 5a:69:19:79:bb:9e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-04-24T23:49:37.889 INFO:teuthology.orchestra.run.smithi156.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-04-24T23:49:37.890 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:37.890 INFO:teuthology.orchestra.run.smithi156.stdout: inet6 fe80::5869:19ff:fe79:bb9e/64 scope link 2024-04-24T23:49:37.896 INFO:teuthology.orchestra.run.smithi156.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:49:37.897 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-04-24T23:49:37.897 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:37.898 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-04-24T23:49:37.898 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-04-24T23:49:37.898 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-04-24T23:49:37.898 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-04-24T23:49:37.898 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-04-24T23:49:37.898 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-04-24T23:49:37.898 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-04-24T23:49:37.985 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:37 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:38.039 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:38.045 DEBUG:teuthology.orchestra.run.smithi156:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:49:38.045 DEBUG:teuthology.orchestra.run.smithi156:> set -e 2024-04-24T23:49:38.046 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set brx.1 up 2024-04-24T23:49:38.046 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link set dev brx.1 master ceph-brx 2024-04-24T23:49:38.046 DEBUG:teuthology.orchestra.run.smithi156:> ') 2024-04-24T23:49:38.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:38 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:49:38.115 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:38.121 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-04-24T23:49:38.121 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:49:38.121 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-04-24T23:49:38.136 INFO:teuthology.orchestra.run.smithi156.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-04-24T23:49:38.136 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:49:38.136 DEBUG:teuthology.orchestra.run.smithi156:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-04-24T23:49:38.191 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-04-24T23:49:38.255 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:49:38.311 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-04-24T23:49:38.311 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-04-24T23:49:38.311 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-04-24T23:49:38.312 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-04-24T23:49:38.312 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-04-24T23:49:38.312 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-04-24T23:49:38.312 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-04-24T23:49:38.312 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-04-24T23:49:38.312 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-04-24T23:49:38.312 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-04-24T23:49:38.312 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-04-24T23:49:38.312 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-04-24T23:49:38.312 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:49:38.312 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-04-24T23:49:38.312 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-04-24T23:49:38.312 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-04-24T23:49:38.312 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysusers.service 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-04-24T23:49:38.313 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T23:49:38.314 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T23:49:38.314 INFO:teuthology.orchestra.run.smithi156.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-24T23:49:38.314 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-24T23:49:38.314 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-24T23:49:38.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:38.314 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-04-24T23:49:38.366 INFO:teuthology.orchestra.run.smithi156.stdout:42 2024-04-24T23:49:38.366 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [42] 2024-04-24T23:49:38.366 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-04-24T23:49:38.408 DEBUG:teuthology.orchestra.run.smithi156:> sudo modprobe fuse 2024-04-24T23:49:38.432 DEBUG:teuthology.orchestra.run.smithi156:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:49:38.489 INFO:teuthology.orchestra.run.smithi156.stdout:/proc 2024-04-24T23:49:38.489 INFO:teuthology.orchestra.run.smithi156.stdout:/sys 2024-04-24T23:49:38.489 INFO:teuthology.orchestra.run.smithi156.stdout:/dev 2024-04-24T23:49:38.489 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/security 2024-04-24T23:49:38.489 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/shm 2024-04-24T23:49:38.489 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/pts 2024-04-24T23:49:38.489 INFO:teuthology.orchestra.run.smithi156.stdout:/run 2024-04-24T23:49:38.489 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/cgroup 2024-04-24T23:49:38.489 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/pstore 2024-04-24T23:49:38.489 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/bpf 2024-04-24T23:49:38.489 INFO:teuthology.orchestra.run.smithi156.stdout:/ 2024-04-24T23:49:38.489 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/selinux 2024-04-24T23:49:38.490 INFO:teuthology.orchestra.run.smithi156.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:49:38.490 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/hugepages 2024-04-24T23:49:38.490 INFO:teuthology.orchestra.run.smithi156.stdout:/dev/mqueue 2024-04-24T23:49:38.490 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/debug 2024-04-24T23:49:38.490 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/tracing 2024-04-24T23:49:38.490 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysusers.service 2024-04-24T23:49:38.490 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/kernel/config 2024-04-24T23:49:38.490 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:49:38.490 INFO:teuthology.orchestra.run.smithi156.stdout:/sys/fs/fuse/connections 2024-04-24T23:49:38.490 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:49:38.490 INFO:teuthology.orchestra.run.smithi156.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:49:38.490 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:49:38.490 INFO:teuthology.orchestra.run.smithi156.stdout:/run/user/1000 2024-04-24T23:49:38.490 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph 2024-04-24T23:49:38.491 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-4 2024-04-24T23:49:38.491 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-5 2024-04-24T23:49:38.491 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-6 2024-04-24T23:49:38.491 INFO:teuthology.orchestra.run.smithi156.stdout:/var/lib/ceph/osd/ceph-7 2024-04-24T23:49:38.491 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns 2024-04-24T23:49:38.491 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T23:49:38.491 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T23:49:38.491 INFO:teuthology.orchestra.run.smithi156.stdout:/home/ubuntu/cephtest/mnt.1 2024-04-24T23:49:38.491 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-24T23:49:38.491 INFO:teuthology.orchestra.run.smithi156.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-24T23:49:38.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:38.492 DEBUG:teuthology.orchestra.run.smithi156:> ls /sys/fs/fuse/connections 2024-04-24T23:49:38.492 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi156.stderr:2024-04-24T23:49:38.490+0000 7f922ba16480 -1 init, newargv = 0x55bb1369e070 newargc=15 2024-04-24T23:49:38.493 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi156.stderr:ceph-fuse[41189]: starting ceph client 2024-04-24T23:49:38.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi156.stderr:ceph-fuse[41189]: starting fuse 2024-04-24T23:49:38.506 INFO:teuthology.orchestra.run.smithi156.stdout:42 2024-04-24T23:49:38.506 INFO:teuthology.orchestra.run.smithi156.stdout:44 2024-04-24T23:49:38.507 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [42, 44] 2024-04-24T23:49:38.507 DEBUG:teuthology.orchestra.run.smithi156:> sudo stdin-killer -- python3 -c ' 2024-04-24T23:49:38.507 DEBUG:teuthology.orchestra.run.smithi156:> import glob 2024-04-24T23:49:38.507 DEBUG:teuthology.orchestra.run.smithi156:> import re 2024-04-24T23:49:38.507 DEBUG:teuthology.orchestra.run.smithi156:> import os 2024-04-24T23:49:38.507 DEBUG:teuthology.orchestra.run.smithi156:> import subprocess 2024-04-24T23:49:38.507 DEBUG:teuthology.orchestra.run.smithi156:> 2024-04-24T23:49:38.507 DEBUG:teuthology.orchestra.run.smithi156:> def _find_admin_socket(client_name): 2024-04-24T23:49:38.507 DEBUG:teuthology.orchestra.run.smithi156:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-04-24T23:49:38.507 DEBUG:teuthology.orchestra.run.smithi156:> files = glob.glob(asok_path) 2024-04-24T23:49:38.507 DEBUG:teuthology.orchestra.run.smithi156:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-04-24T23:49:38.507 DEBUG:teuthology.orchestra.run.smithi156:> 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> # Given a non-glob path, it better be there 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> if "*" not in asok_path: 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> assert(len(files) == 1) 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> return files[0] 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> for f in files: 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> contents = proc_f.read() 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> if mountpoint in contents: 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> return f 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> 2024-04-24T23:49:38.508 DEBUG:teuthology.orchestra.run.smithi156:> print(_find_admin_socket("client.3")) 2024-04-24T23:49:38.509 DEBUG:teuthology.orchestra.run.smithi156:> ' 2024-04-24T23:49:38.624 INFO:teuthology.orchestra.run.smithi156.stdout:/var/run/ceph/ceph-client.3.41189.asok 2024-04-24T23:49:38.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-04-24T23:49:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:49:38.635 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.41189.asok 2024-04-24T23:49:38.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:38.635 DEBUG:teuthology.orchestra.run.smithi156:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.41189.asok status 2024-04-24T23:49:38.767 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-04-24T23:49:38.768 INFO:teuthology.orchestra.run.smithi156.stdout: "metadata": { 2024-04-24T23:49:38.768 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-24T23:49:38.768 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-24T23:49:38.768 INFO:teuthology.orchestra.run.smithi156.stdout: "entity_id": "3", 2024-04-24T23:49:38.768 INFO:teuthology.orchestra.run.smithi156.stdout: "hostname": "smithi156", 2024-04-24T23:49:38.768 INFO:teuthology.orchestra.run.smithi156.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-04-24T23:49:38.768 INFO:teuthology.orchestra.run.smithi156.stdout: "pid": "41189", 2024-04-24T23:49:38.768 INFO:teuthology.orchestra.run.smithi156.stdout: "root": "/" 2024-04-24T23:49:38.768 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-24T23:49:38.768 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_count": 0, 2024-04-24T23:49:38.769 INFO:teuthology.orchestra.run.smithi156.stdout: "dentry_pinned_count": 0, 2024-04-24T23:49:38.769 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 4819, 2024-04-24T23:49:38.769 INFO:teuthology.orchestra.run.smithi156.stdout: "inst": { 2024-04-24T23:49:38.769 INFO:teuthology.orchestra.run.smithi156.stdout: "name": { 2024-04-24T23:49:38.769 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "client", 2024-04-24T23:49:38.769 INFO:teuthology.orchestra.run.smithi156.stdout: "num": 4819 2024-04-24T23:49:38.769 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-24T23:49:38.769 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-04-24T23:49:38.769 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-04-24T23:49:38.769 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.2:0", 2024-04-24T23:49:38.769 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3832586031 2024-04-24T23:49:38.770 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-04-24T23:49:38.770 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-24T23:49:38.770 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": { 2024-04-24T23:49:38.770 INFO:teuthology.orchestra.run.smithi156.stdout: "type": "v1", 2024-04-24T23:49:38.770 INFO:teuthology.orchestra.run.smithi156.stdout: "addr": "192.168.0.2:0", 2024-04-24T23:49:38.770 INFO:teuthology.orchestra.run.smithi156.stdout: "nonce": 3832586031 2024-04-24T23:49:38.770 INFO:teuthology.orchestra.run.smithi156.stdout: }, 2024-04-24T23:49:38.770 INFO:teuthology.orchestra.run.smithi156.stdout: "inst_str": "client.4819 192.168.0.2:0/3832586031", 2024-04-24T23:49:38.770 INFO:teuthology.orchestra.run.smithi156.stdout: "addr_str": "192.168.0.2:0/3832586031", 2024-04-24T23:49:38.770 INFO:teuthology.orchestra.run.smithi156.stdout: "inode_count": 1, 2024-04-24T23:49:38.771 INFO:teuthology.orchestra.run.smithi156.stdout: "mds_epoch": 8, 2024-04-24T23:49:38.771 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch": 27, 2024-04-24T23:49:38.771 INFO:teuthology.orchestra.run.smithi156.stdout: "osd_epoch_barrier": 0, 2024-04-24T23:49:38.771 INFO:teuthology.orchestra.run.smithi156.stdout: "blocklisted": false, 2024-04-24T23:49:38.771 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs" 2024-04-24T23:49:38.771 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-04-24T23:49:38.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:38.776 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-04-24T23:49:38.776 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T23:49:38.796 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-04-24T23:49:38.796 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T23:49:38.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:38.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T23:49:38.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:38.866 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-04-24T23:49:38.866 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.2 2024-04-24T23:49:38.927 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-04-24T23:49:38.927 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-04-24T23:49:38.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:38.927 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-04-24T23:49:38.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:38.997 DEBUG:teuthology.orchestra.run.smithi156:> stat --file-system '--printf=%T 2024-04-24T23:49:38.997 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.1 2024-04-24T23:49:39.013 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2024-04-24T23:49:39.013 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-04-24T23:49:39.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:39.013 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-04-24T23:49:39.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:39.078 DEBUG:teuthology.orchestra.run.smithi156:> stat --file-system '--printf=%T 2024-04-24T23:49:39.078 DEBUG:teuthology.orchestra.run.smithi156:> ' -- /home/ubuntu/cephtest/mnt.3 2024-04-24T23:49:39.134 INFO:teuthology.orchestra.run.smithi156.stdout:fuseblk 2024-04-24T23:49:39.134 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-04-24T23:49:39.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:39.135 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-04-24T23:49:39.199 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-04-24T23:49:39.228 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:39.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.342+0000 7fe52d74f640 1 Processor -- start 2024-04-24T23:49:39.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.342+0000 7fe52d74f640 1 -- start start 2024-04-24T23:49:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.343+0000 7fe52d74f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe528071a00 0x7fe528071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.343+0000 7fe52d74f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe528072320 con 0x7fe5280ce7c0 2024-04-24T23:49:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.343+0000 7fe52d74f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5280cf3a0 con 0x7fe528144740 2024-04-24T23:49:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.343+0000 7fe52d74f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe52815c1f0 con 0x7fe528071a00 2024-04-24T23:49:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.343+0000 7fe5277fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe5280ce7c0 0x7fe5280cd690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52250/0 (socket says 172.21.15.16:52250) 2024-04-24T23:49:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.343+0000 7fe5277fe640 1 -- 172.21.15.16:0/1568736110 learned_addr learned my addr 172.21.15.16:0/1568736110 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.344+0000 7fe526ffd640 1 --2- 172.21.15.16:0/1568736110 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe528071a00 0x7fe528071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.344+0000 7fe525ffb640 1 -- 172.21.15.16:0/1568736110 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 940511074 0 0) 0x7fe528072320 con 0x7fe5280ce7c0 2024-04-24T23:49:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.344+0000 7fe525ffb640 1 -- 172.21.15.16:0/1568736110 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe510005730 con 0x7fe5280ce7c0 2024-04-24T23:49:39.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.344+0000 7fe525ffb640 1 -- 172.21.15.16:0/1568736110 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3595826560 0 0) 0x7fe5280cf3a0 con 0x7fe528144740 2024-04-24T23:49:39.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.344+0000 7fe525ffb640 1 -- 172.21.15.16:0/1568736110 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe528072320 con 0x7fe528144740 2024-04-24T23:49:39.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.344+0000 7fe525ffb640 1 -- 172.21.15.16:0/1568736110 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 297067997 0 0) 0x7fe510005730 con 0x7fe5280ce7c0 2024-04-24T23:49:39.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.344+0000 7fe526ffd640 1 -- 172.21.15.16:0/1568736110 >> v1:172.21.15.156:6789/0 conn(0x7fe528144740 legacy=0x7fe52815c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.344+0000 7fe526ffd640 1 -- 172.21.15.16:0/1568736110 >> v1:172.21.15.16:6789/0 conn(0x7fe5280ce7c0 legacy=0x7fe5280cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.344+0000 7fe526ffd640 1 -- 172.21.15.16:0/1568736110 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5280cc8d0 con 0x7fe528071a00 2024-04-24T23:49:39.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.345+0000 7fe526ffd640 1 --2- 172.21.15.16:0/1568736110 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe528071a00 0x7fe528071de0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe5200097d0 tx=0x7fe52002a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8a68d9bba9edfc1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:39.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.345+0000 7fe525ffb640 1 -- 172.21.15.16:0/1568736110 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe520037070 con 0x7fe528071a00 2024-04-24T23:49:39.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.345+0000 7fe525ffb640 1 -- 172.21.15.16:0/1568736110 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe52002ac30 con 0x7fe528071a00 2024-04-24T23:49:39.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.345+0000 7fe525ffb640 1 -- 172.21.15.16:0/1568736110 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5180019f0 con 0x7fe528071a00 2024-04-24T23:49:39.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.346+0000 7fe52d74f640 1 -- 172.21.15.16:0/1568736110 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe528071a00 msgr2=0x7fe528071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.346+0000 7fe52d74f640 1 --2- 172.21.15.16:0/1568736110 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe528071a00 0x7fe528071de0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe5200097d0 tx=0x7fe52002a1a0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.346+0000 7fe52d74f640 1 -- 172.21.15.16:0/1568736110 shutdown_connections 2024-04-24T23:49:39.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.346+0000 7fe52d74f640 1 --2- 172.21.15.16:0/1568736110 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe528071a00 0x7fe528071de0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.346+0000 7fe52d74f640 1 -- 172.21.15.16:0/1568736110 >> 172.21.15.16:0/1568736110 conn(0x7fe5280b2000 msgr2=0x7fe5280cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:39.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.346+0000 7fe52d74f640 1 -- 172.21.15.16:0/1568736110 shutdown_connections 2024-04-24T23:49:39.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.346+0000 7fe52d74f640 1 -- 172.21.15.16:0/1568736110 wait complete. 2024-04-24T23:49:39.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.347+0000 7fe52d74f640 1 Processor -- start 2024-04-24T23:49:39.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.347+0000 7fe52d74f640 1 -- start start 2024-04-24T23:49:39.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe52d74f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe528071a00 0x7fe5280cd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe52d74f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe5280ce7c0 0x7fe5281e3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe52d74f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5281e39b0 0x7fe52813a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe52d74f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe5280cd170 con 0x7fe5281e39b0 2024-04-24T23:49:39.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe52d74f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe5280cce90 con 0x7fe528071a00 2024-04-24T23:49:39.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe52d74f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe52815cab0 con 0x7fe5280ce7c0 2024-04-24T23:49:39.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe5277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5281e39b0 0x7fe52813a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe5277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5281e39b0 0x7fe52813a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59610/0 (socket says 172.21.15.16:59610) 2024-04-24T23:49:39.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe5277fe640 1 -- 172.21.15.16:0/1221310729 learned_addr learned my addr 172.21.15.16:0/1221310729 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:39.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe526ffd640 1 --2- 172.21.15.16:0/1221310729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe528071a00 0x7fe5280cd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe5267fc640 1 --2- 172.21.15.16:0/1221310729 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe5280ce7c0 0x7fe5281e3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe5277fe640 1 -- 172.21.15.16:0/1221310729 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe5280ce7c0 msgr2=0x7fe5281e3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe5277fe640 1 --2- 172.21.15.16:0/1221310729 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe5280ce7c0 0x7fe5281e3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe5277fe640 1 -- 172.21.15.16:0/1221310729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe528071a00 msgr2=0x7fe5280cd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe5277fe640 1 --2- 172.21.15.16:0/1221310729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe528071a00 0x7fe5280cd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.348+0000 7fe5277fe640 1 -- 172.21.15.16:0/1221310729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5281e3f10 con 0x7fe5281e39b0 2024-04-24T23:49:39.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.349+0000 7fe5277fe640 1 --2- 172.21.15.16:0/1221310729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5281e39b0 0x7fe52813a030 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe518001350 tx=0x7fe51800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:39.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.349+0000 7fe50f7fe640 1 -- 172.21.15.16:0/1221310729 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518017070 con 0x7fe5281e39b0 2024-04-24T23:49:39.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.349+0000 7fe50f7fe640 1 -- 172.21.15.16:0/1221310729 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe518012400 con 0x7fe5281e39b0 2024-04-24T23:49:39.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.349+0000 7fe52d74f640 1 -- 172.21.15.16:0/1221310729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5281e41f0 con 0x7fe5281e39b0 2024-04-24T23:49:39.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.349+0000 7fe50f7fe640 1 -- 172.21.15.16:0/1221310729 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe518012830 con 0x7fe5281e39b0 2024-04-24T23:49:39.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.349+0000 7fe52d74f640 1 -- 172.21.15.16:0/1221310729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe52814c3d0 con 0x7fe5281e39b0 2024-04-24T23:49:39.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.350+0000 7fe52d74f640 1 -- 172.21.15.16:0/1221310729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5280731f0 con 0x7fe5281e39b0 2024-04-24T23:49:39.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.350+0000 7fe50f7fe640 1 -- 172.21.15.16:0/1221310729 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe51801a070 con 0x7fe5281e39b0 2024-04-24T23:49:39.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.351+0000 7fe50f7fe640 1 --2- 172.21.15.16:0/1221310729 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe51406be30 0x7fe51406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.351+0000 7fe50f7fe640 1 -- 172.21.15.16:0/1221310729 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe51808b210 con 0x7fe5281e39b0 2024-04-24T23:49:39.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.351+0000 7fe526ffd640 1 --2- 172.21.15.16:0/1221310729 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe51406be30 0x7fe51406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.351+0000 7fe526ffd640 1 --2- 172.21.15.16:0/1221310729 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe51406be30 0x7fe51406e2f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe520034000 tx=0x7fe52002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:39.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.355+0000 7fe50f7fe640 1 -- 172.21.15.16:0/1221310729 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe5180597c0 con 0x7fe5281e39b0 2024-04-24T23:49:39.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.541+0000 7fe52d74f640 1 -- 172.21.15.16:0/1221310729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe52804f5e0 con 0x7fe5281e39b0 2024-04-24T23:49:39.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.542+0000 7fe50f7fe640 1 -- 172.21.15.16:0/1221310729 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7fe5180590c0 con 0x7fe5281e39b0 2024-04-24T23:49:39.542 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:39.542 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:39.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.544+0000 7fe52d74f640 1 -- 172.21.15.16:0/1221310729 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe51406be30 msgr2=0x7fe51406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.544+0000 7fe52d74f640 1 --2- 172.21.15.16:0/1221310729 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe51406be30 0x7fe51406e2f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe520034000 tx=0x7fe52002e000 comp rx=0 tx=0).stop 2024-04-24T23:49:39.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.544+0000 7fe52d74f640 1 -- 172.21.15.16:0/1221310729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5281e39b0 msgr2=0x7fe52813a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.544+0000 7fe52d74f640 1 --2- 172.21.15.16:0/1221310729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5281e39b0 0x7fe52813a030 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe518001350 tx=0x7fe51800cf90 comp rx=0 tx=0).stop 2024-04-24T23:49:39.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.544+0000 7fe52d74f640 1 -- 172.21.15.16:0/1221310729 shutdown_connections 2024-04-24T23:49:39.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.544+0000 7fe52d74f640 1 --2- 172.21.15.16:0/1221310729 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe51406be30 0x7fe51406e2f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.544+0000 7fe52d74f640 1 --2- 172.21.15.16:0/1221310729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe5281e39b0 0x7fe52813a030 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.544+0000 7fe52d74f640 1 --2- 172.21.15.16:0/1221310729 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe5280ce7c0 0x7fe5281e3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.544+0000 7fe52d74f640 1 --2- 172.21.15.16:0/1221310729 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe528071a00 0x7fe5280cd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.544+0000 7fe52d74f640 1 -- 172.21.15.16:0/1221310729 >> 172.21.15.16:0/1221310729 conn(0x7fe5280b2000 msgr2=0x7fe52815b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:39.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.544+0000 7fe52d74f640 1 -- 172.21.15.16:0/1221310729 shutdown_connections 2024-04-24T23:49:39.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.544+0000 7fe52d74f640 1 -- 172.21.15.16:0/1221310729 wait complete. 2024-04-24T23:49:39.555 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:39.555 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:39.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.660+0000 7fc4c84fc640 1 Processor -- start 2024-04-24T23:49:39.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.660+0000 7fc4c84fc640 1 -- start start 2024-04-24T23:49:39.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c84fc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc4c00cd790 0x7fc4c00d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c84fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4c015e5c0 0x7fc4c00d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c84fc640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4c015ec30 con 0x7fc4c00d12d0 2024-04-24T23:49:39.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c84fc640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fc4c015f4d0 con 0x7fc4c00cd790 2024-04-24T23:49:39.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c84fc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fc4c015ef10 con 0x7fc4c015e5c0 2024-04-24T23:49:39.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c6298640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc4c00d12d0 0x7fc4c00c8580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52254/0 (socket says 172.21.15.16:52254) 2024-04-24T23:49:39.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c6298640 1 -- 172.21.15.16:0/402271962 learned_addr learned my addr 172.21.15.16:0/402271962 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:39.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c5296640 1 -- 172.21.15.16:0/402271962 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3855251943 0 0) 0x7fc4c015ec30 con 0x7fc4c00d12d0 2024-04-24T23:49:39.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c6a99640 1 --2- 172.21.15.16:0/402271962 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4c015e5c0 0x7fc4c00d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c5a97640 1 --2- 172.21.15.16:0/402271962 >> v2:172.21.15.156:3300/0 conn(0x7fc4c00cd790 0x7fc4c00d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c5296640 1 -- 172.21.15.16:0/402271962 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4ac003520 con 0x7fc4c00d12d0 2024-04-24T23:49:39.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c5296640 1 -- 172.21.15.16:0/402271962 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1975336138 0 0) 0x7fc4ac003520 con 0x7fc4c00d12d0 2024-04-24T23:49:39.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c5296640 1 -- 172.21.15.16:0/402271962 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4c015e5c0 msgr2=0x7fc4c00d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:39.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c5296640 1 --2- 172.21.15.16:0/402271962 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4c015e5c0 0x7fc4c00d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c5296640 1 -- 172.21.15.16:0/402271962 >> v2:172.21.15.156:3300/0 conn(0x7fc4c00cd790 msgr2=0x7fc4c00d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:39.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c5296640 1 --2- 172.21.15.16:0/402271962 >> v2:172.21.15.156:3300/0 conn(0x7fc4c00cd790 0x7fc4c00d16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c5296640 1 -- 172.21.15.16:0/402271962 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4c015f1f0 con 0x7fc4c00d12d0 2024-04-24T23:49:39.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.661+0000 7fc4c5296640 1 -- 172.21.15.16:0/402271962 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7fc4bc001670 con 0x7fc4c00d12d0 2024-04-24T23:49:39.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c5296640 1 -- 172.21.15.16:0/402271962 >> v1:172.21.15.16:6789/0 conn(0x7fc4c00d12d0 legacy=0x7fc4c00c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c5296640 1 --2- 172.21.15.16:0/402271962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4ac0039b0 0x7fc4ac003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c5296640 1 --2- 172.21.15.16:0/402271962 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4c015e5c0 0x7fc4ac004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c5296640 1 --2- 172.21.15.16:0/402271962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4ac004d40 0x7fc4ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c5296640 1 -- 172.21.15.16:0/402271962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c015f4d0 con 0x7fc4ac004d40 2024-04-24T23:49:39.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c5296640 1 -- 172.21.15.16:0/402271962 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c015ef10 con 0x7fc4ac0039b0 2024-04-24T23:49:39.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c5296640 1 -- 172.21.15.16:0/402271962 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fc4ac00ba40 con 0x7fc4c015e5c0 2024-04-24T23:49:39.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c6a99640 1 --2- 172.21.15.16:0/402271962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4ac004d40 0x7fc4ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c5a97640 1 --2- 172.21.15.16:0/402271962 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4c015e5c0 0x7fc4ac004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c6298640 1 --2- 172.21.15.16:0/402271962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4ac0039b0 0x7fc4ac003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c6a99640 1 -- 172.21.15.16:0/402271962 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4c015e5c0 msgr2=0x7fc4ac004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:39.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c6a99640 1 --2- 172.21.15.16:0/402271962 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4c015e5c0 0x7fc4ac004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c6a99640 1 -- 172.21.15.16:0/402271962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4ac0039b0 msgr2=0x7fc4ac003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:39.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c6a99640 1 --2- 172.21.15.16:0/402271962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4ac0039b0 0x7fc4ac003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c6a99640 1 -- 172.21.15.16:0/402271962 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4ac003520 con 0x7fc4ac004d40 2024-04-24T23:49:39.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.662+0000 7fc4c6298640 1 --2- 172.21.15.16:0/402271962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4ac0039b0 0x7fc4ac003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:39.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.663+0000 7fc4c6a99640 1 --2- 172.21.15.16:0/402271962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4ac004d40 0x7fc4ac0051c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8009fc0 tx=0x7fc4ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e36f725cc5c01faa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:39.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.663+0000 7fc4c5296640 1 -- 172.21.15.16:0/402271962 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8009c10 con 0x7fc4ac004d40 2024-04-24T23:49:39.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.663+0000 7fc4c5296640 1 -- 172.21.15.16:0/402271962 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4b8036070 con 0x7fc4ac004d40 2024-04-24T23:49:39.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.663+0000 7fc4c5296640 1 -- 172.21.15.16:0/402271962 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8031880 con 0x7fc4ac004d40 2024-04-24T23:49:39.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.663+0000 7fc4c84fc640 1 -- 172.21.15.16:0/402271962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4ac004d40 msgr2=0x7fc4ac0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.663+0000 7fc4c84fc640 1 --2- 172.21.15.16:0/402271962 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4ac004d40 0x7fc4ac0051c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fc4b8009fc0 tx=0x7fc4ac005830 comp rx=0 tx=0).stop 2024-04-24T23:49:39.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.663+0000 7fc4c6a99640 1 -- 172.21.15.16:0/402271962 reap_dead start 2024-04-24T23:49:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.664+0000 7fc4c84fc640 1 -- 172.21.15.16:0/402271962 shutdown_connections 2024-04-24T23:49:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.664+0000 7fc4c84fc640 1 -- 172.21.15.16:0/402271962 >> 172.21.15.16:0/402271962 conn(0x7fc4c0072e70 msgr2=0x7fc4c0073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.664+0000 7fc4c84fc640 1 -- 172.21.15.16:0/402271962 shutdown_connections 2024-04-24T23:49:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.664+0000 7fc4c84fc640 1 -- 172.21.15.16:0/402271962 wait complete. 2024-04-24T23:49:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.665+0000 7fc4c84fc640 1 Processor -- start 2024-04-24T23:49:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.665+0000 7fc4c84fc640 1 -- start start 2024-04-24T23:49:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.665+0000 7fc4c84fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4ac004d40 0x7fc4c01519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.665+0000 7fc4c84fc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4c015f680 0x7fc4c015fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.665+0000 7fc4c84fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c00b48e0 0x7fc4c00b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.665+0000 7fc4c84fc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c00d22b0 con 0x7fc4c00b48e0 2024-04-24T23:49:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.665+0000 7fc4c84fc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc4c00d2130 con 0x7fc4c015f680 2024-04-24T23:49:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.665+0000 7fc4c84fc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fc4c00d2430 con 0x7fc4ac004d40 2024-04-24T23:49:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4c6a99640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c00b48e0 0x7fc4c00b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4c6a99640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c00b48e0 0x7fc4c00b4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59626/0 (socket says 172.21.15.16:59626) 2024-04-24T23:49:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4c6a99640 1 -- 172.21.15.16:0/1410016855 learned_addr learned my addr 172.21.15.16:0/1410016855 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4c6298640 1 --2- 172.21.15.16:0/1410016855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4ac004d40 0x7fc4c01519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4c5a97640 1 --2- 172.21.15.16:0/1410016855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4c015f680 0x7fc4c015fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4c6a99640 1 -- 172.21.15.16:0/1410016855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4ac004d40 msgr2=0x7fc4c01519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4c6a99640 1 --2- 172.21.15.16:0/1410016855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4ac004d40 0x7fc4c01519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4c6a99640 1 -- 172.21.15.16:0/1410016855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4c015f680 msgr2=0x7fc4c015fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4c6a99640 1 --2- 172.21.15.16:0/1410016855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4c015f680 0x7fc4c015fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4c6a99640 1 -- 172.21.15.16:0/1410016855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4c00b55a0 con 0x7fc4c00b48e0 2024-04-24T23:49:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4c6a99640 1 --2- 172.21.15.16:0/1410016855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c00b48e0 0x7fc4c00b4d60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800e230 tx=0x7fc4b8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4b6ffd640 1 -- 172.21.15.16:0/1410016855 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc007d50 con 0x7fc4c00b48e0 2024-04-24T23:49:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4b6ffd640 1 -- 172.21.15.16:0/1410016855 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4b8031dd0 con 0x7fc4c00b48e0 2024-04-24T23:49:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4c84fc640 1 -- 172.21.15.16:0/1410016855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4c00b5880 con 0x7fc4c00b48e0 2024-04-24T23:49:39.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4b6ffd640 1 -- 172.21.15.16:0/1410016855 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8009c10 con 0x7fc4c00b48e0 2024-04-24T23:49:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.666+0000 7fc4c84fc640 1 -- 172.21.15.16:0/1410016855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4c00b5e40 con 0x7fc4c00b48e0 2024-04-24T23:49:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.667+0000 7fc4c84fc640 1 -- 172.21.15.16:0/1410016855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4c0068f90 con 0x7fc4c00b48e0 2024-04-24T23:49:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.668+0000 7fc4b6ffd640 1 -- 172.21.15.16:0/1410016855 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc4b8005d10 con 0x7fc4c00b48e0 2024-04-24T23:49:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.668+0000 7fc4b6ffd640 1 --2- 172.21.15.16:0/1410016855 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc4a406be60 0x7fc4a406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.668+0000 7fc4b6ffd640 1 -- 172.21.15.16:0/1410016855 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc4b80ac3f0 con 0x7fc4c00b48e0 2024-04-24T23:49:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.669+0000 7fc4c6298640 1 --2- 172.21.15.16:0/1410016855 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc4a406be60 0x7fc4a406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.669+0000 7fc4c6298640 1 --2- 172.21.15.16:0/1410016855 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc4a406be60 0x7fc4a406e320 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc001120 tx=0x7fc4bc0078f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:39.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.673+0000 7fc4b6ffd640 1 -- 172.21.15.16:0/1410016855 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc4b804a060 con 0x7fc4c00b48e0 2024-04-24T23:49:39.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.854+0000 7fc4c84fc640 1 -- 172.21.15.16:0/1410016855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc4c00b0b40 con 0x7fc4c00b48e0 2024-04-24T23:49:39.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.854+0000 7fc4b6ffd640 1 -- 172.21.15.16:0/1410016855 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fc4b807a9a0 con 0x7fc4c00b48e0 2024-04-24T23:49:39.854 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.856+0000 7fc4c84fc640 1 -- 172.21.15.16:0/1410016855 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc4a406be60 msgr2=0x7fc4a406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.856+0000 7fc4c84fc640 1 --2- 172.21.15.16:0/1410016855 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc4a406be60 0x7fc4a406e320 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc001120 tx=0x7fc4bc0078f0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.856+0000 7fc4c84fc640 1 -- 172.21.15.16:0/1410016855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c00b48e0 msgr2=0x7fc4c00b4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.856+0000 7fc4c84fc640 1 --2- 172.21.15.16:0/1410016855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c00b48e0 0x7fc4c00b4d60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800e230 tx=0x7fc4b8009750 comp rx=0 tx=0).stop 2024-04-24T23:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.856+0000 7fc4c84fc640 1 -- 172.21.15.16:0/1410016855 shutdown_connections 2024-04-24T23:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.856+0000 7fc4c84fc640 1 --2- 172.21.15.16:0/1410016855 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc4a406be60 0x7fc4a406e320 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.856+0000 7fc4c84fc640 1 --2- 172.21.15.16:0/1410016855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4c00b48e0 0x7fc4c00b4d60 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.856+0000 7fc4c84fc640 1 --2- 172.21.15.16:0/1410016855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4c015f680 0x7fc4c015fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.856+0000 7fc4c84fc640 1 --2- 172.21.15.16:0/1410016855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4ac004d40 0x7fc4c01519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.856+0000 7fc4c84fc640 1 -- 172.21.15.16:0/1410016855 >> 172.21.15.16:0/1410016855 conn(0x7fc4c0072e70 msgr2=0x7fc4c01622f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.856+0000 7fc4c84fc640 1 -- 172.21.15.16:0/1410016855 shutdown_connections 2024-04-24T23:49:39.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.856+0000 7fc4c84fc640 1 -- 172.21.15.16:0/1410016855 wait complete. 2024-04-24T23:49:39.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:39.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.971+0000 7f847944d640 1 Processor -- start 2024-04-24T23:49:39.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.971+0000 7f847944d640 1 -- start start 2024-04-24T23:49:39.971 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.971+0000 7f847944d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84740d12d0 0x7f84740d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.971+0000 7f847944d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f84740cd790 0x7f847415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.971+0000 7f847944d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f847415ec30 con 0x7f84740c8580 2024-04-24T23:49:39.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.971+0000 7f847944d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f847415f4d0 con 0x7f84740cd790 2024-04-24T23:49:39.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.971+0000 7f847944d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f847415ef10 con 0x7f84740d12d0 2024-04-24T23:49:39.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.972+0000 7f84727fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f84740cd790 0x7f847415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.972+0000 7f84727fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f84740cd790 0x7f847415e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59636/0 (socket says 172.21.15.16:59636) 2024-04-24T23:49:39.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.972+0000 7f84727fc640 1 -- 172.21.15.16:0/4294194431 learned_addr learned my addr 172.21.15.16:0/4294194431 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:39.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.972+0000 7f8472ffd640 1 --2- 172.21.15.16:0/4294194431 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84740d12d0 0x7f84740d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.972+0000 7f84727fc640 1 -- 172.21.15.16:0/4294194431 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84740d12d0 msgr2=0x7f84740d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:39.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.972+0000 7f84727fc640 1 --2- 172.21.15.16:0/4294194431 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84740d12d0 0x7f84740d16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.972+0000 7f84727fc640 1 -- 172.21.15.16:0/4294194431 >> v1:172.21.15.156:6789/0 conn(0x7f84740c8580 legacy=0x7f84740d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:39.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.972+0000 7f84727fc640 1 -- 172.21.15.16:0/4294194431 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f847415f1f0 con 0x7f84740cd790 2024-04-24T23:49:39.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.973+0000 7f84727fc640 1 --2- 172.21.15.16:0/4294194431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84740cd790 0x7f847415e5c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f84740c6aa0 tx=0x7f846802a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66be9329b8667821 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:39.973 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.973+0000 7f8471ffb640 1 -- 172.21.15.16:0/4294194431 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8468037070 con 0x7f84740cd790 2024-04-24T23:49:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.973+0000 7f8471ffb640 1 -- 172.21.15.16:0/4294194431 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f846802adc0 con 0x7f84740cd790 2024-04-24T23:49:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.973+0000 7f8471ffb640 1 -- 172.21.15.16:0/4294194431 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8468032660 con 0x7f84740cd790 2024-04-24T23:49:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.973+0000 7f847944d640 1 -- 172.21.15.16:0/4294194431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84740cd790 msgr2=0x7f847415e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.973+0000 7f847944d640 1 --2- 172.21.15.16:0/4294194431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84740cd790 0x7f847415e5c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f84740c6aa0 tx=0x7f846802a0b0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.973+0000 7f847944d640 1 -- 172.21.15.16:0/4294194431 shutdown_connections 2024-04-24T23:49:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.973+0000 7f847944d640 1 --2- 172.21.15.16:0/4294194431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84740cd790 0x7f847415e5c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.973+0000 7f847944d640 1 --2- 172.21.15.16:0/4294194431 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84740d12d0 0x7f84740d16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.973+0000 7f847944d640 1 -- 172.21.15.16:0/4294194431 >> 172.21.15.16:0/4294194431 conn(0x7f8474072e70 msgr2=0x7f8474073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.973+0000 7f847944d640 1 -- 172.21.15.16:0/4294194431 shutdown_connections 2024-04-24T23:49:39.974 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.973+0000 7f847944d640 1 -- 172.21.15.16:0/4294194431 wait complete. 2024-04-24T23:49:39.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.975+0000 7f847944d640 1 Processor -- start 2024-04-24T23:49:39.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.975+0000 7f847944d640 1 -- start start 2024-04-24T23:49:39.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.975+0000 7f847944d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84740c8580 0x7f84740b6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.976 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.975+0000 7f847944d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84740cd790 0x7f84741f72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.975+0000 7f847944d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84740d12d0 0x7f84741f0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.975+0000 7f847944d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f84740d2840 con 0x7f84740c8580 2024-04-24T23:49:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.975+0000 7f847944d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f84740c89f0 con 0x7f84740d12d0 2024-04-24T23:49:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.975+0000 7f847944d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f84740d29c0 con 0x7f84740cd790 2024-04-24T23:49:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.975+0000 7f8472ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84740c8580 0x7f84740b6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f8472ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84740c8580 0x7f84740b6130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59640/0 (socket says 172.21.15.16:59640) 2024-04-24T23:49:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f8472ffd640 1 -- 172.21.15.16:0/3299841669 learned_addr learned my addr 172.21.15.16:0/3299841669 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f84727fc640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84740cd790 0x7f84741f72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f84737fe640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84740d12d0 0x7f84741f0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f8472ffd640 1 -- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84740cd790 msgr2=0x7f84741f72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f8472ffd640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84740cd790 0x7f84741f72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f8472ffd640 1 -- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84740d12d0 msgr2=0x7f84741f0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f8472ffd640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84740d12d0 0x7f84741f0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f8472ffd640 1 -- 172.21.15.16:0/3299841669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84741564e0 con 0x7f84740c8580 2024-04-24T23:49:39.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f84727fc640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84740cd790 0x7f84741f72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:39.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f8472ffd640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84740c8580 0x7f84740b6130 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f846400d800 tx=0x7f846400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:39.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f845b7fe640 1 -- 172.21.15.16:0/3299841669 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846400b6c0 con 0x7f84740c8580 2024-04-24T23:49:39.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f845b7fe640 1 -- 172.21.15.16:0/3299841669 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8464019070 con 0x7f84740c8580 2024-04-24T23:49:39.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f847944d640 1 -- 172.21.15.16:0/3299841669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84741567c0 con 0x7f84740c8580 2024-04-24T23:49:39.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f845b7fe640 1 -- 172.21.15.16:0/3299841669 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f846400ba80 con 0x7f84740c8580 2024-04-24T23:49:39.978 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.976+0000 7f847944d640 1 -- 172.21.15.16:0/3299841669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8474156d80 con 0x7f84740c8580 2024-04-24T23:49:39.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.977+0000 7f847944d640 1 -- 172.21.15.16:0/3299841669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8474068f90 con 0x7f84740c8580 2024-04-24T23:49:39.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.978+0000 7f845b7fe640 1 -- 172.21.15.16:0/3299841669 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f846400bc20 con 0x7f84740c8580 2024-04-24T23:49:39.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.978+0000 7f845b7fe640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f846006be30 0x7f846006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:39.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.978+0000 7f845b7fe640 1 -- 172.21.15.16:0/3299841669 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f846408b640 con 0x7f84740c8580 2024-04-24T23:49:39.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.978+0000 7f84727fc640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f846006be30 0x7f846006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:39.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.979+0000 7f84727fc640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f846006be30 0x7f846006e2f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8468034000 tx=0x7f846802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:39.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:39.983+0000 7f845b7fe640 1 -- 172.21.15.16:0/3299841669 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8464059510 con 0x7f84740c8580 2024-04-24T23:49:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.163+0000 7f847944d640 1 -- 172.21.15.16:0/3299841669 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f84740b6b30 con 0x7f84740c8580 2024-04-24T23:49:40.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.164+0000 7f845b7fe640 1 -- 172.21.15.16:0/3299841669 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f846405d810 con 0x7f84740c8580 2024-04-24T23:49:40.164 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:40.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.166+0000 7f847944d640 1 -- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f846006be30 msgr2=0x7f846006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.166+0000 7f847944d640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f846006be30 0x7f846006e2f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f8468034000 tx=0x7f846802e000 comp rx=0 tx=0).stop 2024-04-24T23:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.166+0000 7f847944d640 1 -- 172.21.15.16:0/3299841669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84740c8580 msgr2=0x7f84740b6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.166+0000 7f847944d640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84740c8580 0x7f84740b6130 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f846400d800 tx=0x7f846400dcc0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.166+0000 7f847944d640 1 -- 172.21.15.16:0/3299841669 shutdown_connections 2024-04-24T23:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.166+0000 7f847944d640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f846006be30 0x7f846006e2f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.166+0000 7f847944d640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f84740d12d0 0x7f84741f0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.166+0000 7f847944d640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f84740cd790 0x7f84741f72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.166+0000 7f847944d640 1 --2- 172.21.15.16:0/3299841669 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84740c8580 0x7f84740b6130 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.166+0000 7f847944d640 1 -- 172.21.15.16:0/3299841669 >> 172.21.15.16:0/3299841669 conn(0x7f8474072e70 msgr2=0x7f84740cdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.166+0000 7f847944d640 1 -- 172.21.15.16:0/3299841669 shutdown_connections 2024-04-24T23:49:40.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.166+0000 7f847944d640 1 -- 172.21.15.16:0/3299841669 wait complete. 2024-04-24T23:49:40.176 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.280+0000 7f1f54cfc640 1 Processor -- start 2024-04-24T23:49:40.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.280+0000 7f1f54cfc640 1 -- start start 2024-04-24T23:49:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.281+0000 7f1f54cfc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f500d12d0 0x7f1f500d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.281+0000 7f1f54cfc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f5015e5c0 0x7f1f500d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.281+0000 7f1f54cfc640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f5015ec30 con 0x7f1f500cd790 2024-04-24T23:49:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.281+0000 7f1f54cfc640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1f5015f4d0 con 0x7f1f500d12d0 2024-04-24T23:49:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.281+0000 7f1f54cfc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1f5015ef10 con 0x7f1f5015e5c0 2024-04-24T23:49:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.281+0000 7f1f4e59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f500d12d0 0x7f1f500d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.281+0000 7f1f4e59c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f500d12d0 0x7f1f500d16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59650/0 (socket says 172.21.15.16:59650) 2024-04-24T23:49:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.281+0000 7f1f4e59c640 1 -- 172.21.15.16:0/2140208036 learned_addr learned my addr 172.21.15.16:0/2140208036 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.281+0000 7f1f4ed9d640 1 --2- 172.21.15.16:0/2140208036 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f5015e5c0 0x7f1f500d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.281+0000 7f1f4e59c640 1 -- 172.21.15.16:0/2140208036 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f5015e5c0 msgr2=0x7f1f500d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:40.281 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.281+0000 7f1f4e59c640 1 --2- 172.21.15.16:0/2140208036 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f5015e5c0 0x7f1f500d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.281+0000 7f1f4e59c640 1 -- 172.21.15.16:0/2140208036 >> v1:172.21.15.156:6789/0 conn(0x7f1f500cd790 legacy=0x7f1f500c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.282 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.281+0000 7f1f4e59c640 1 -- 172.21.15.16:0/2140208036 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f5015f1f0 con 0x7f1f500d12d0 2024-04-24T23:49:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.282+0000 7f1f4e59c640 1 --2- 172.21.15.16:0/2140208036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f500d12d0 0x7f1f500d16b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1f5015f650 tx=0x7f1f4002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fdda5a0a1d5b311 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.282+0000 7f1f4d59a640 1 -- 172.21.15.16:0/2140208036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f40037070 con 0x7f1f500d12d0 2024-04-24T23:49:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.282+0000 7f1f4d59a640 1 -- 172.21.15.16:0/2140208036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f40032440 con 0x7f1f500d12d0 2024-04-24T23:49:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.282+0000 7f1f4d59a640 1 -- 172.21.15.16:0/2140208036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f40032810 con 0x7f1f500d12d0 2024-04-24T23:49:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.283+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2140208036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f500d12d0 msgr2=0x7f1f500d16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.283+0000 7f1f54cfc640 1 --2- 172.21.15.16:0/2140208036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f500d12d0 0x7f1f500d16b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f1f5015f650 tx=0x7f1f4002a130 comp rx=0 tx=0).stop 2024-04-24T23:49:40.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.283+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2140208036 shutdown_connections 2024-04-24T23:49:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.283+0000 7f1f54cfc640 1 --2- 172.21.15.16:0/2140208036 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f5015e5c0 0x7f1f500d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.283+0000 7f1f54cfc640 1 --2- 172.21.15.16:0/2140208036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f500d12d0 0x7f1f500d16b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.283+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2140208036 >> 172.21.15.16:0/2140208036 conn(0x7f1f50072e70 msgr2=0x7f1f50073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.283+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2140208036 shutdown_connections 2024-04-24T23:49:40.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.283+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2140208036 wait complete. 2024-04-24T23:49:40.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.284+0000 7f1f54cfc640 1 Processor -- start 2024-04-24T23:49:40.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.284+0000 7f1f54cfc640 1 -- start start 2024-04-24T23:49:40.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f54cfc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f500cd790 0x7f1f500ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f54cfc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f500d12d0 0x7f1f50153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f54cfc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f5015e5c0 0x7f1f50157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f54cfc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1f500d27f0 con 0x7f1f500cd790 2024-04-24T23:49:40.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f54cfc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1f500d2670 con 0x7f1f500d12d0 2024-04-24T23:49:40.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f54cfc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1f500d2970 con 0x7f1f5015e5c0 2024-04-24T23:49:40.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f4e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f500cd790 0x7f1f500ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f4e59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f500cd790 0x7f1f500ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59656/0 (socket says 172.21.15.16:59656) 2024-04-24T23:49:40.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f4e59c640 1 -- 172.21.15.16:0/2731812745 learned_addr learned my addr 172.21.15.16:0/2731812745 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:40.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f4dd9b640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f500d12d0 0x7f1f50153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f4ed9d640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f5015e5c0 0x7f1f50157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f4e59c640 1 -- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f5015e5c0 msgr2=0x7f1f50157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f4e59c640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f5015e5c0 0x7f1f50157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f4e59c640 1 -- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f500d12d0 msgr2=0x7f1f50153b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f4e59c640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f500d12d0 0x7f1f50153b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f4e59c640 1 -- 172.21.15.16:0/2731812745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f500adfe0 con 0x7f1f500cd790 2024-04-24T23:49:40.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.285+0000 7f1f4dd9b640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f500d12d0 0x7f1f50153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:40.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.286+0000 7f1f4e59c640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f500cd790 0x7f1f500ad8e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1f4002a170 tx=0x7f1f400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:40.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.286+0000 7f1f3effd640 1 -- 172.21.15.16:0/2731812745 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f40037040 con 0x7f1f500cd790 2024-04-24T23:49:40.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.286+0000 7f1f3effd640 1 -- 172.21.15.16:0/2731812745 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f40042070 con 0x7f1f500cd790 2024-04-24T23:49:40.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.286+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2731812745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f500ae2c0 con 0x7f1f500cd790 2024-04-24T23:49:40.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.286+0000 7f1f3effd640 1 -- 172.21.15.16:0/2731812745 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f40030920 con 0x7f1f500cd790 2024-04-24T23:49:40.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.286+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2731812745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f500ae880 con 0x7f1f500cd790 2024-04-24T23:49:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.287+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2731812745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f50068f90 con 0x7f1f500cd790 2024-04-24T23:49:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.287+0000 7f1f3effd640 1 -- 172.21.15.16:0/2731812745 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1f40030ac0 con 0x7f1f500cd790 2024-04-24T23:49:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.288+0000 7f1f3effd640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1f3006be30 0x7f1f3006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.288+0000 7f1f3effd640 1 -- 172.21.15.16:0/2731812745 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1f400abb80 con 0x7f1f500cd790 2024-04-24T23:49:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.288+0000 7f1f4dd9b640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1f3006be30 0x7f1f3006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.289+0000 7f1f4dd9b640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1f3006be30 0x7f1f3006e2f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1f500af980 tx=0x7f1f44009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:40.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.292+0000 7f1f3effd640 1 -- 172.21.15.16:0/2731812745 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1f40079b70 con 0x7f1f500cd790 2024-04-24T23:49:40.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.476+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2731812745 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1f501608d0 con 0x7f1f500cd790 2024-04-24T23:49:40.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.476+0000 7f1f3effd640 1 -- 172.21.15.16:0/2731812745 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f1f4007de70 con 0x7f1f500cd790 2024-04-24T23:49:40.476 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.478+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1f3006be30 msgr2=0x7f1f3006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.478+0000 7f1f54cfc640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1f3006be30 0x7f1f3006e2f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1f500af980 tx=0x7f1f44009040 comp rx=0 tx=0).stop 2024-04-24T23:49:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.478+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2731812745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f500cd790 msgr2=0x7f1f500ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.478+0000 7f1f54cfc640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f500cd790 0x7f1f500ad8e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1f4002a170 tx=0x7f1f400046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.478+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2731812745 shutdown_connections 2024-04-24T23:49:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.478+0000 7f1f54cfc640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1f3006be30 0x7f1f3006e2f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.478+0000 7f1f54cfc640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1f5015e5c0 0x7f1f50157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.478+0000 7f1f54cfc640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f500d12d0 0x7f1f50153b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.478+0000 7f1f54cfc640 1 --2- 172.21.15.16:0/2731812745 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f500cd790 0x7f1f500ad8e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.478+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2731812745 >> 172.21.15.16:0/2731812745 conn(0x7f1f50072e70 msgr2=0x7f1f50073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:40.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.478+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2731812745 shutdown_connections 2024-04-24T23:49:40.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.478+0000 7f1f54cfc640 1 -- 172.21.15.16:0/2731812745 wait complete. 2024-04-24T23:49:40.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:40.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.592+0000 7ff219cfc640 1 Processor -- start 2024-04-24T23:49:40.655 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.592+0000 7ff219cfc640 1 -- start start 2024-04-24T23:49:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.593+0000 7ff219cfc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff2140d12d0 0x7ff2140d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.593+0000 7ff219cfc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff2140cd790 0x7ff21415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.593+0000 7ff219cfc640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff21415ec30 con 0x7ff2140c8580 2024-04-24T23:49:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.593+0000 7ff219cfc640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff21415f4d0 con 0x7ff2140d12d0 2024-04-24T23:49:40.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.593+0000 7ff219cfc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff21415ef10 con 0x7ff2140cd790 2024-04-24T23:49:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.593+0000 7ff2137fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff2140d12d0 0x7ff2140d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.593+0000 7ff213fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff2140c8580 0x7ff2140d2130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52268/0 (socket says 172.21.15.16:52268) 2024-04-24T23:49:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.593+0000 7ff213fff640 1 -- 172.21.15.16:0/638952821 learned_addr learned my addr 172.21.15.16:0/638952821 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.593+0000 7ff2137fe640 1 -- 172.21.15.16:0/638952821 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff2140cd790 msgr2=0x7ff21415e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.593+0000 7ff212ffd640 1 --2- 172.21.15.16:0/638952821 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff2140cd790 0x7ff21415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.593+0000 7ff2127fc640 1 -- 172.21.15.16:0/638952821 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1698210149 0 0) 0x7ff21415ec30 con 0x7ff2140c8580 2024-04-24T23:49:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.594+0000 7ff2137fe640 1 --2- 172.21.15.16:0/638952821 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff2140cd790 0x7ff21415e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.594+0000 7ff2137fe640 1 -- 172.21.15.16:0/638952821 >> v1:172.21.15.16:6789/0 conn(0x7ff2140c8580 legacy=0x7ff2140d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.594+0000 7ff2137fe640 1 -- 172.21.15.16:0/638952821 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff21415f1f0 con 0x7ff2140d12d0 2024-04-24T23:49:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.594+0000 7ff2137fe640 1 --2- 172.21.15.16:0/638952821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2140d12d0 0x7ff2140d16b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff2040097a0 tx=0x7ff20402a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=599bfc44b92f9d76 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.594+0000 7ff2127fc640 1 -- 172.21.15.16:0/638952821 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff204037070 con 0x7ff2140d12d0 2024-04-24T23:49:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.594+0000 7ff2127fc640 1 -- 172.21.15.16:0/638952821 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff20402adb0 con 0x7ff2140d12d0 2024-04-24T23:49:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.594+0000 7ff2127fc640 1 -- 172.21.15.16:0/638952821 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff204032660 con 0x7ff2140d12d0 2024-04-24T23:49:40.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.595+0000 7ff219cfc640 1 -- 172.21.15.16:0/638952821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2140d12d0 msgr2=0x7ff2140d16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.595+0000 7ff219cfc640 1 --2- 172.21.15.16:0/638952821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2140d12d0 0x7ff2140d16b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff2040097a0 tx=0x7ff20402a060 comp rx=0 tx=0).stop 2024-04-24T23:49:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.595+0000 7ff219cfc640 1 -- 172.21.15.16:0/638952821 shutdown_connections 2024-04-24T23:49:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.595+0000 7ff219cfc640 1 --2- 172.21.15.16:0/638952821 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff2140cd790 0x7ff21415e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.595+0000 7ff219cfc640 1 --2- 172.21.15.16:0/638952821 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2140d12d0 0x7ff2140d16b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.595+0000 7ff219cfc640 1 -- 172.21.15.16:0/638952821 >> 172.21.15.16:0/638952821 conn(0x7ff214072e70 msgr2=0x7ff214073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.595+0000 7ff219cfc640 1 -- 172.21.15.16:0/638952821 shutdown_connections 2024-04-24T23:49:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.595+0000 7ff219cfc640 1 -- 172.21.15.16:0/638952821 wait complete. 2024-04-24T23:49:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.596+0000 7ff219cfc640 1 Processor -- start 2024-04-24T23:49:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.596+0000 7ff219cfc640 1 -- start start 2024-04-24T23:49:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff219cfc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2140c8580 0x7ff2140b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff219cfc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff2140cd790 0x7ff214155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff219cfc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2140d12d0 0x7ff214156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff219cfc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff2140d2840 con 0x7ff2140c8580 2024-04-24T23:49:40.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff219cfc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff2140c8a10 con 0x7ff2140d12d0 2024-04-24T23:49:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff219cfc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff2140d29c0 con 0x7ff2140cd790 2024-04-24T23:49:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff2137fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2140c8580 0x7ff2140b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff2137fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2140c8580 0x7ff2140b2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59680/0 (socket says 172.21.15.16:59680) 2024-04-24T23:49:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff2137fe640 1 -- 172.21.15.16:0/462455859 learned_addr learned my addr 172.21.15.16:0/462455859 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff212ffd640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff2140cd790 0x7ff214155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff213fff640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2140d12d0 0x7ff214156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff2137fe640 1 -- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff2140cd790 msgr2=0x7ff214155550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff2137fe640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff2140cd790 0x7ff214155550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff2137fe640 1 -- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2140d12d0 msgr2=0x7ff214156dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff2137fe640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2140d12d0 0x7ff214156dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff2137fe640 1 -- 172.21.15.16:0/462455859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff214155a90 con 0x7ff2140c8580 2024-04-24T23:49:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.597+0000 7ff213fff640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2140d12d0 0x7ff214156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.598+0000 7ff2137fe640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2140c8580 0x7ff2140b2d20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff20402a650 tx=0x7ff2040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:40.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.598+0000 7ff1fbfff640 1 -- 172.21.15.16:0/462455859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff204041020 con 0x7ff2140c8580 2024-04-24T23:49:40.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.598+0000 7ff1fbfff640 1 -- 172.21.15.16:0/462455859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff20403c870 con 0x7ff2140c8580 2024-04-24T23:49:40.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.598+0000 7ff219cfc640 1 -- 172.21.15.16:0/462455859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff214155d30 con 0x7ff2140c8580 2024-04-24T23:49:40.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.598+0000 7ff1fbfff640 1 -- 172.21.15.16:0/462455859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff204037040 con 0x7ff2140c8580 2024-04-24T23:49:40.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.598+0000 7ff219cfc640 1 -- 172.21.15.16:0/462455859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2141562f0 con 0x7ff2140c8580 2024-04-24T23:49:40.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.599+0000 7ff219cfc640 1 -- 172.21.15.16:0/462455859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff214068f90 con 0x7ff2140c8580 2024-04-24T23:49:40.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.599+0000 7ff1fbfff640 1 -- 172.21.15.16:0/462455859 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff204044020 con 0x7ff2140c8580 2024-04-24T23:49:40.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.600+0000 7ff1fbfff640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff20006be30 0x7ff20006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.600+0000 7ff1fbfff640 1 -- 172.21.15.16:0/462455859 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ff2040abae0 con 0x7ff2140c8580 2024-04-24T23:49:40.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.600+0000 7ff212ffd640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff20006be30 0x7ff20006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.601+0000 7ff212ffd640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff20006be30 0x7ff20006e2f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff208006fd0 tx=0x7ff208008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:40.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.604+0000 7ff1fbfff640 1 -- 172.21.15.16:0/462455859 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff20404b030 con 0x7ff2140c8580 2024-04-24T23:49:40.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.788+0000 7ff219cfc640 1 -- 172.21.15.16:0/462455859 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff21415e8b0 con 0x7ff2140c8580 2024-04-24T23:49:40.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.788+0000 7ff1fbfff640 1 -- 172.21.15.16:0/462455859 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7ff20407a090 con 0x7ff2140c8580 2024-04-24T23:49:40.788 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:40.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.790+0000 7ff219cfc640 1 -- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff20006be30 msgr2=0x7ff20006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.790+0000 7ff219cfc640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff20006be30 0x7ff20006e2f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff208006fd0 tx=0x7ff208008040 comp rx=0 tx=0).stop 2024-04-24T23:49:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.790+0000 7ff219cfc640 1 -- 172.21.15.16:0/462455859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2140c8580 msgr2=0x7ff2140b2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.790+0000 7ff219cfc640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2140c8580 0x7ff2140b2d20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff20402a650 tx=0x7ff2040046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.790+0000 7ff219cfc640 1 -- 172.21.15.16:0/462455859 shutdown_connections 2024-04-24T23:49:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.790+0000 7ff219cfc640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff20006be30 0x7ff20006e2f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.790+0000 7ff219cfc640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2140d12d0 0x7ff214156dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.790+0000 7ff219cfc640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff2140cd790 0x7ff214155550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.790+0000 7ff219cfc640 1 --2- 172.21.15.16:0/462455859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2140c8580 0x7ff2140b2d20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.790+0000 7ff219cfc640 1 -- 172.21.15.16:0/462455859 >> 172.21.15.16:0/462455859 conn(0x7ff214072e70 msgr2=0x7ff2140d2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.791+0000 7ff219cfc640 1 -- 172.21.15.16:0/462455859 shutdown_connections 2024-04-24T23:49:40.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.791+0000 7ff219cfc640 1 -- 172.21.15.16:0/462455859 wait complete. 2024-04-24T23:49:40.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:40.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.906+0000 7efee6ea5640 1 Processor -- start 2024-04-24T23:49:40.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.906+0000 7efee6ea5640 1 -- start start 2024-04-24T23:49:40.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efee6ea5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efee0144740 0x7efee015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efee6ea5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7efee015c8e0 0x7efee00ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efee6ea5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efee00cf3a0 con 0x7efee0071a00 2024-04-24T23:49:40.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efee6ea5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7efee015c1f0 con 0x7efee0144740 2024-04-24T23:49:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efee6ea5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7efee00cc8d0 con 0x7efee015c8e0 2024-04-24T23:49:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efedffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efee0144740 0x7efee015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efee4c41640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efee0071a00 0x7efee0071de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52270/0 (socket says 172.21.15.16:52270) 2024-04-24T23:49:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efedffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efee0144740 0x7efee015c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59688/0 (socket says 172.21.15.16:59688) 2024-04-24T23:49:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efee4c41640 1 -- 172.21.15.16:0/862838923 learned_addr learned my addr 172.21.15.16:0/862838923 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efee5442640 1 --2- 172.21.15.16:0/862838923 >> v2:172.21.15.156:3300/0 conn(0x7efee015c8e0 0x7efee00ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efedffff640 1 -- 172.21.15.16:0/862838923 >> v2:172.21.15.156:3300/0 conn(0x7efee015c8e0 msgr2=0x7efee00ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efedffff640 1 --2- 172.21.15.16:0/862838923 >> v2:172.21.15.156:3300/0 conn(0x7efee015c8e0 0x7efee00ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efedffff640 1 -- 172.21.15.16:0/862838923 >> v1:172.21.15.16:6789/0 conn(0x7efee0071a00 legacy=0x7efee0071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efedf7fe640 1 -- 172.21.15.16:0/862838923 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4179176836 0 0) 0x7efee00cf3a0 con 0x7efee0071a00 2024-04-24T23:49:40.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.907+0000 7efedffff640 1 -- 172.21.15.16:0/862838923 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee00ccbb0 con 0x7efee0144740 2024-04-24T23:49:40.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.908+0000 7efedffff640 1 --2- 172.21.15.16:0/862838923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efee0144740 0x7efee015c3a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efee00af640 tx=0x7efed402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b09f079470179f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.908+0000 7efedf7fe640 1 -- 172.21.15.16:0/862838923 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed4037070 con 0x7efee0144740 2024-04-24T23:49:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.908+0000 7efedf7fe640 1 -- 172.21.15.16:0/862838923 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efed4032440 con 0x7efee0144740 2024-04-24T23:49:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.908+0000 7efedf7fe640 1 -- 172.21.15.16:0/862838923 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed4032810 con 0x7efee0144740 2024-04-24T23:49:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.909+0000 7efee6ea5640 1 -- 172.21.15.16:0/862838923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efee0144740 msgr2=0x7efee015c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.909+0000 7efee6ea5640 1 --2- 172.21.15.16:0/862838923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efee0144740 0x7efee015c3a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7efee00af640 tx=0x7efed402a1a0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.909+0000 7efee6ea5640 1 -- 172.21.15.16:0/862838923 shutdown_connections 2024-04-24T23:49:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.909+0000 7efee6ea5640 1 --2- 172.21.15.16:0/862838923 >> v2:172.21.15.156:3300/0 conn(0x7efee015c8e0 0x7efee00ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.909+0000 7efee6ea5640 1 --2- 172.21.15.16:0/862838923 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efee0144740 0x7efee015c3a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.909+0000 7efee6ea5640 1 -- 172.21.15.16:0/862838923 >> 172.21.15.16:0/862838923 conn(0x7efee00b2000 msgr2=0x7efee00cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.909+0000 7efee6ea5640 1 -- 172.21.15.16:0/862838923 shutdown_connections 2024-04-24T23:49:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.909+0000 7efee6ea5640 1 -- 172.21.15.16:0/862838923 wait complete. 2024-04-24T23:49:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.910+0000 7efee6ea5640 1 Processor -- start 2024-04-24T23:49:40.910 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.910+0000 7efee6ea5640 1 -- start start 2024-04-24T23:49:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee6ea5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efee0071a00 0x7efee0141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee6ea5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee015c8e0 0x7efee00aea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee6ea5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efee0141650 0x7efee00aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee6ea5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efee00ced00 con 0x7efee0141650 2024-04-24T23:49:40.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee6ea5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efee00cd170 con 0x7efee015c8e0 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee6ea5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7efee00cee80 con 0x7efee0071a00 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee5442640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efee0141650 0x7efee00aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee5442640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efee0141650 0x7efee00aefc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59700/0 (socket says 172.21.15.16:59700) 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee5442640 1 -- 172.21.15.16:0/885680912 learned_addr learned my addr 172.21.15.16:0/885680912 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee4c41640 1 --2- 172.21.15.16:0/885680912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efee0071a00 0x7efee0141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efedffff640 1 --2- 172.21.15.16:0/885680912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee015c8e0 0x7efee00aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee5442640 1 -- 172.21.15.16:0/885680912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efee0071a00 msgr2=0x7efee0141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee5442640 1 --2- 172.21.15.16:0/885680912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efee0071a00 0x7efee0141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee5442640 1 -- 172.21.15.16:0/885680912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee015c8e0 msgr2=0x7efee00aea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee5442640 1 --2- 172.21.15.16:0/885680912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee015c8e0 0x7efee00aea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.911+0000 7efee5442640 1 -- 172.21.15.16:0/885680912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee0141bd0 con 0x7efee0141650 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.912+0000 7efee5442640 1 --2- 172.21.15.16:0/885680912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efee0141650 0x7efee00aefc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efed000d5a0 tx=0x7efed000da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.912+0000 7efedd7fa640 1 -- 172.21.15.16:0/885680912 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed00125f0 con 0x7efee0141650 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.912+0000 7efedd7fa640 1 -- 172.21.15.16:0/885680912 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efed0019070 con 0x7efee0141650 2024-04-24T23:49:40.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.912+0000 7efedd7fa640 1 -- 172.21.15.16:0/885680912 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efed00129f0 con 0x7efee0141650 2024-04-24T23:49:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.912+0000 7efee6ea5640 1 -- 172.21.15.16:0/885680912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee0141eb0 con 0x7efee0141650 2024-04-24T23:49:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.912+0000 7efee6ea5640 1 -- 172.21.15.16:0/885680912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee0142470 con 0x7efee0141650 2024-04-24T23:49:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.913+0000 7efee6ea5640 1 -- 172.21.15.16:0/885680912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efee00731f0 con 0x7efee0141650 2024-04-24T23:49:40.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.913+0000 7efedd7fa640 1 -- 172.21.15.16:0/885680912 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7efed0012b90 con 0x7efee0141650 2024-04-24T23:49:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.913+0000 7efedd7fa640 1 --2- 172.21.15.16:0/885680912 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efecc06be60 0x7efecc06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.914+0000 7efedd7fa640 1 -- 172.21.15.16:0/885680912 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7efed008b520 con 0x7efee0141650 2024-04-24T23:49:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.914+0000 7efee4c41640 1 --2- 172.21.15.16:0/885680912 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efecc06be60 0x7efecc06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:40.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.914+0000 7efee4c41640 1 --2- 172.21.15.16:0/885680912 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efecc06be60 0x7efecc06e320 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efed80012e0 tx=0x7efed8007ed0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:40.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:40.918+0000 7efedd7fa640 1 -- 172.21.15.16:0/885680912 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7efed0059ae0 con 0x7efee0141650 2024-04-24T23:49:41.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.102+0000 7efee6ea5640 1 -- 172.21.15.16:0/885680912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efee004f5e0 con 0x7efee0141650 2024-04-24T23:49:41.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.104+0000 7efedd7fa640 1 -- 172.21.15.16:0/885680912 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7efed00593e0 con 0x7efee0141650 2024-04-24T23:49:41.103 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:41.104 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:41.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.105+0000 7efee6ea5640 1 -- 172.21.15.16:0/885680912 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efecc06be60 msgr2=0x7efecc06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.105+0000 7efee6ea5640 1 --2- 172.21.15.16:0/885680912 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efecc06be60 0x7efecc06e320 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7efed80012e0 tx=0x7efed8007ed0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.105+0000 7efee6ea5640 1 -- 172.21.15.16:0/885680912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efee0141650 msgr2=0x7efee00aefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.105+0000 7efee6ea5640 1 --2- 172.21.15.16:0/885680912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efee0141650 0x7efee00aefc0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7efed000d5a0 tx=0x7efed000da60 comp rx=0 tx=0).stop 2024-04-24T23:49:41.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.106+0000 7efee6ea5640 1 -- 172.21.15.16:0/885680912 shutdown_connections 2024-04-24T23:49:41.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.106+0000 7efee6ea5640 1 --2- 172.21.15.16:0/885680912 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efecc06be60 0x7efecc06e320 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.106+0000 7efee6ea5640 1 --2- 172.21.15.16:0/885680912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efee0141650 0x7efee00aefc0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.106+0000 7efee6ea5640 1 --2- 172.21.15.16:0/885680912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee015c8e0 0x7efee00aea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.106+0000 7efee6ea5640 1 --2- 172.21.15.16:0/885680912 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efee0071a00 0x7efee0141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.106+0000 7efee6ea5640 1 -- 172.21.15.16:0/885680912 >> 172.21.15.16:0/885680912 conn(0x7efee00b2000 msgr2=0x7efee0142f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:41.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.106+0000 7efee6ea5640 1 -- 172.21.15.16:0/885680912 shutdown_connections 2024-04-24T23:49:41.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.106+0000 7efee6ea5640 1 -- 172.21.15.16:0/885680912 wait complete. 2024-04-24T23:49:41.116 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:41.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.222+0000 7f2202c4d640 1 Processor -- start 2024-04-24T23:49:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.222+0000 7f2202c4d640 1 -- start start 2024-04-24T23:49:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f2202c4d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21fc0d2260 0x7f21fc0b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f2202c4d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f21fc0b21d0 0x7f21fc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f2202c4d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21fc0b2a10 con 0x7f21fc1478d0 2024-04-24T23:49:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f2202c4d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f21fc0b3620 con 0x7f21fc0b21d0 2024-04-24T23:49:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f2202c4d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f21fc0b2cf0 con 0x7f21fc0d2260 2024-04-24T23:49:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f22009e9640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f21fc1478d0 0x7f21fc0d1b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52272/0 (socket says 172.21.15.16:52272) 2024-04-24T23:49:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f22009e9640 1 -- 172.21.15.16:0/2115423585 learned_addr learned my addr 172.21.15.16:0/2115423585 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f22011ea640 1 --2- 172.21.15.16:0/2115423585 >> v2:172.21.15.156:3300/0 conn(0x7f21fc0b21d0 0x7f21fc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f21f3fff640 1 --2- 172.21.15.16:0/2115423585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21fc0d2260 0x7f21fc0b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f21f37fe640 1 -- 172.21.15.16:0/2115423585 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2077982893 0 0) 0x7f21fc0b2a10 con 0x7f21fc1478d0 2024-04-24T23:49:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f21f37fe640 1 -- 172.21.15.16:0/2115423585 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21e4003520 con 0x7f21fc1478d0 2024-04-24T23:49:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f21f37fe640 1 -- 172.21.15.16:0/2115423585 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4050888164 0 0) 0x7f21e4003520 con 0x7f21fc1478d0 2024-04-24T23:49:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f21f37fe640 1 -- 172.21.15.16:0/2115423585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21fc0d2260 msgr2=0x7f21fc0b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f21f37fe640 1 --2- 172.21.15.16:0/2115423585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21fc0d2260 0x7f21fc0b1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f21f37fe640 1 -- 172.21.15.16:0/2115423585 >> v2:172.21.15.156:3300/0 conn(0x7f21fc0b21d0 msgr2=0x7f21fc0b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f21f37fe640 1 --2- 172.21.15.16:0/2115423585 >> v2:172.21.15.156:3300/0 conn(0x7f21fc0b21d0 0x7f21fc0b4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f21f37fe640 1 -- 172.21.15.16:0/2115423585 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21fc0b2fd0 con 0x7f21fc1478d0 2024-04-24T23:49:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f21f37fe640 1 -- 172.21.15.16:0/2115423585 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f21f8001670 con 0x7f21fc1478d0 2024-04-24T23:49:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f21f37fe640 1 -- 172.21.15.16:0/2115423585 >> v1:172.21.15.16:6789/0 conn(0x7f21fc1478d0 legacy=0x7f21fc0d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.223+0000 7f21f37fe640 1 --2- 172.21.15.16:0/2115423585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21fc0d2260 0x7f21e40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.224+0000 7f21f37fe640 1 --2- 172.21.15.16:0/2115423585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21e4003f30 0x7f21e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.224+0000 7f21f37fe640 1 --2- 172.21.15.16:0/2115423585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21e4004d40 0x7f21e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.224+0000 7f21f37fe640 1 -- 172.21.15.16:0/2115423585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc0b3620 con 0x7f21e4003f30 2024-04-24T23:49:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.224+0000 7f21f37fe640 1 -- 172.21.15.16:0/2115423585 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc0b2cf0 con 0x7f21e4004d40 2024-04-24T23:49:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.224+0000 7f21f37fe640 1 -- 172.21.15.16:0/2115423585 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f21e400b9c0 con 0x7f21fc0d2260 2024-04-24T23:49:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.224+0000 7f21f3fff640 1 --2- 172.21.15.16:0/2115423585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21e4003f30 0x7f21e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.224+0000 7f22009e9640 1 --2- 172.21.15.16:0/2115423585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21fc0d2260 0x7f21e40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.224+0000 7f22011ea640 1 --2- 172.21.15.16:0/2115423585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21e4004d40 0x7f21e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.224+0000 7f21f3fff640 1 -- 172.21.15.16:0/2115423585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21fc0d2260 msgr2=0x7f21e40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.224+0000 7f21f3fff640 1 --2- 172.21.15.16:0/2115423585 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21fc0d2260 0x7f21e40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.224+0000 7f21f3fff640 1 -- 172.21.15.16:0/2115423585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21e4004d40 msgr2=0x7f21e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.224+0000 7f21f3fff640 1 --2- 172.21.15.16:0/2115423585 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21e4004d40 0x7f21e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.224+0000 7f21f3fff640 1 -- 172.21.15.16:0/2115423585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21e4003520 con 0x7f21e4003f30 2024-04-24T23:49:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.225+0000 7f21f3fff640 1 --2- 172.21.15.16:0/2115423585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21e4003f30 0x7f21e4004800 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009ac0 tx=0x7f21e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2098958f80a8278 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.225+0000 7f21f37fe640 1 -- 172.21.15.16:0/2115423585 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec036070 con 0x7f21e4003f30 2024-04-24T23:49:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.225+0000 7f21f37fe640 1 -- 172.21.15.16:0/2115423585 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21ec031400 con 0x7f21e4003f30 2024-04-24T23:49:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.225+0000 7f21f37fe640 1 -- 172.21.15.16:0/2115423585 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21ec009690 con 0x7f21e4003f30 2024-04-24T23:49:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.225+0000 7f2202c4d640 1 -- 172.21.15.16:0/2115423585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21e4003f30 msgr2=0x7f21e4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.225+0000 7f2202c4d640 1 --2- 172.21.15.16:0/2115423585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21e4003f30 0x7f21e4004800 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f21ec009ac0 tx=0x7f21e4005830 comp rx=0 tx=0).stop 2024-04-24T23:49:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.225+0000 7f22011ea640 1 -- 172.21.15.16:0/2115423585 reap_dead start 2024-04-24T23:49:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.225+0000 7f2202c4d640 1 -- 172.21.15.16:0/2115423585 shutdown_connections 2024-04-24T23:49:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.225+0000 7f2202c4d640 1 -- 172.21.15.16:0/2115423585 >> 172.21.15.16:0/2115423585 conn(0x7f21fc06b2c0 msgr2=0x7f21fc072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.225+0000 7f2202c4d640 1 -- 172.21.15.16:0/2115423585 shutdown_connections 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.225+0000 7f2202c4d640 1 -- 172.21.15.16:0/2115423585 wait complete. 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.227+0000 7f2202c4d640 1 Processor -- start 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.227+0000 7f2202c4d640 1 -- start start 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.227+0000 7f2202c4d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21e4003f30 0x7f21fc15d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.227+0000 7f2202c4d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21fc13d250 0x7f21fc13d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.227+0000 7f2202c4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21fc15e690 0x7f21fc15eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.227+0000 7f2202c4d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc0b2660 con 0x7f21fc15e690 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.227+0000 7f2202c4d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f21fc0b1c90 con 0x7f21fc13d250 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.227+0000 7f2202c4d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f21fc13f760 con 0x7f21e4003f30 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f22011ea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21fc15e690 0x7f21fc15eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f22011ea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21fc15e690 0x7f21fc15eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59730/0 (socket says 172.21.15.16:59730) 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f22011ea640 1 -- 172.21.15.16:0/25096819 learned_addr learned my addr 172.21.15.16:0/25096819 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f21f3fff640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21fc13d250 0x7f21fc13d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f22009e9640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21e4003f30 0x7f21fc15d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f22011ea640 1 -- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21e4003f30 msgr2=0x7f21fc15d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f22011ea640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21e4003f30 0x7f21fc15d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f22011ea640 1 -- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21fc13d250 msgr2=0x7f21fc13d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f22011ea640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21fc13d250 0x7f21fc13d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f22011ea640 1 -- 172.21.15.16:0/25096819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21fc15f410 con 0x7f21fc15e690 2024-04-24T23:49:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f22009e9640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21e4003f30 0x7f21fc15d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f21f3fff640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21fc13d250 0x7f21fc13d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f22011ea640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21fc15e690 0x7f21fc15eb10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0cfd50 tx=0x7f21f400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f21f17fa640 1 -- 172.21.15.16:0/25096819 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f400fde0 con 0x7f21fc15e690 2024-04-24T23:49:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f21f17fa640 1 -- 172.21.15.16:0/25096819 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21f4015430 con 0x7f21fc15e690 2024-04-24T23:49:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f21f17fa640 1 -- 172.21.15.16:0/25096819 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21f4015820 con 0x7f21fc15e690 2024-04-24T23:49:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f2202c4d640 1 -- 172.21.15.16:0/25096819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21fc15f6f0 con 0x7f21fc15e690 2024-04-24T23:49:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.228+0000 7f2202c4d640 1 -- 172.21.15.16:0/25096819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21fc15fcb0 con 0x7f21fc15e690 2024-04-24T23:49:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.229+0000 7f2202c4d640 1 -- 172.21.15.16:0/25096819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21fc068f90 con 0x7f21fc15e690 2024-04-24T23:49:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.230+0000 7f21f17fa640 1 -- 172.21.15.16:0/25096819 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f21f40159c0 con 0x7f21fc15e690 2024-04-24T23:49:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.230+0000 7f21f17fa640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f21e806be30 0x7f21e806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.230+0000 7f21f17fa640 1 -- 172.21.15.16:0/25096819 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f21f408c100 con 0x7f21fc15e690 2024-04-24T23:49:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.230+0000 7f22009e9640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f21e806be30 0x7f21e806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.231+0000 7f22009e9640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f21e806be30 0x7f21e806e2f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f21f8001eb0 tx=0x7f21f8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:41.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.234+0000 7f21f17fa640 1 -- 172.21.15.16:0/25096819 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f21f4019030 con 0x7f21fc15e690 2024-04-24T23:49:41.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.418+0000 7f2202c4d640 1 -- 172.21.15.16:0/25096819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f21fc13f530 con 0x7f21fc15e690 2024-04-24T23:49:41.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.418+0000 7f21f17fa640 1 -- 172.21.15.16:0/25096819 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f21f4010040 con 0x7f21fc15e690 2024-04-24T23:49:41.418 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:41.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.420+0000 7f2202c4d640 1 -- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f21e806be30 msgr2=0x7f21e806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.420+0000 7f2202c4d640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f21e806be30 0x7f21e806e2f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f21f8001eb0 tx=0x7f21f8009040 comp rx=0 tx=0).stop 2024-04-24T23:49:41.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.420+0000 7f2202c4d640 1 -- 172.21.15.16:0/25096819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21fc15e690 msgr2=0x7f21fc15eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.420+0000 7f2202c4d640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21fc15e690 0x7f21fc15eb10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f21fc0cfd50 tx=0x7f21f400f580 comp rx=0 tx=0).stop 2024-04-24T23:49:41.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.420+0000 7f2202c4d640 1 -- 172.21.15.16:0/25096819 shutdown_connections 2024-04-24T23:49:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.420+0000 7f2202c4d640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f21e806be30 0x7f21e806e2f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.420+0000 7f2202c4d640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21fc15e690 0x7f21fc15eb10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.420+0000 7f2202c4d640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f21fc13d250 0x7f21fc13d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.420+0000 7f2202c4d640 1 --2- 172.21.15.16:0/25096819 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f21e4003f30 0x7f21fc15d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.420+0000 7f2202c4d640 1 -- 172.21.15.16:0/25096819 >> 172.21.15.16:0/25096819 conn(0x7f21fc06b2c0 msgr2=0x7f21fc0b5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.420+0000 7f2202c4d640 1 -- 172.21.15.16:0/25096819 shutdown_connections 2024-04-24T23:49:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.420+0000 7f2202c4d640 1 -- 172.21.15.16:0/25096819 wait complete. 2024-04-24T23:49:41.431 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:41.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.535+0000 7fa99ab70640 1 Processor -- start 2024-04-24T23:49:41.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.535+0000 7fa99ab70640 1 -- start start 2024-04-24T23:49:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.536+0000 7fa99ab70640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa99406d360 0x7fa99406deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.536+0000 7fa99ab70640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99406d8d0 con 0x7fa9940af980 2024-04-24T23:49:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.536+0000 7fa99ab70640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa99406dbb0 con 0x7fa99415dff0 2024-04-24T23:49:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.536+0000 7fa99ab70640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa9940732d0 con 0x7fa99406d360 2024-04-24T23:49:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.536+0000 7fa993fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa9940af980 0x7fa994071e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52274/0 (socket says 172.21.15.16:52274) 2024-04-24T23:49:41.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.536+0000 7fa993fff640 1 -- 172.21.15.16:0/3159388915 learned_addr learned my addr 172.21.15.16:0/3159388915 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:41.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.536+0000 7fa99910d640 1 --2- 172.21.15.16:0/3159388915 >> v2:172.21.15.156:3300/0 conn(0x7fa99406d360 0x7fa99406deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.536+0000 7fa9937fe640 1 -- 172.21.15.16:0/3159388915 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 209105857 0 0) 0x7fa99406d8d0 con 0x7fa9940af980 2024-04-24T23:49:41.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.536+0000 7fa9937fe640 1 -- 172.21.15.16:0/3159388915 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa984003520 con 0x7fa9940af980 2024-04-24T23:49:41.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.536+0000 7fa9937fe640 1 -- 172.21.15.16:0/3159388915 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3276858242 0 0) 0x7fa984003520 con 0x7fa9940af980 2024-04-24T23:49:41.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.536+0000 7fa99910d640 1 -- 172.21.15.16:0/3159388915 >> v1:172.21.15.156:6789/0 conn(0x7fa99415dff0 legacy=0x7fa9940af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.536+0000 7fa99910d640 1 -- 172.21.15.16:0/3159388915 >> v1:172.21.15.16:6789/0 conn(0x7fa9940af980 legacy=0x7fa994071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.537+0000 7fa99910d640 1 -- 172.21.15.16:0/3159388915 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa994071330 con 0x7fa99406d360 2024-04-24T23:49:41.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.537+0000 7fa99910d640 1 --2- 172.21.15.16:0/3159388915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa99406d360 0x7fa99406deb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa98c009d90 tx=0x7fa98c02a680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23ff89112b01fa23 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:41.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.537+0000 7fa9937fe640 1 -- 172.21.15.16:0/3159388915 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa980001670 con 0x7fa99406d360 2024-04-24T23:49:41.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.538+0000 7fa9937fe640 1 -- 172.21.15.16:0/3159388915 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa98c007540 con 0x7fa99406d360 2024-04-24T23:49:41.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.538+0000 7fa9937fe640 1 -- 172.21.15.16:0/3159388915 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98c0078f0 con 0x7fa99406d360 2024-04-24T23:49:41.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.538+0000 7fa99ab70640 1 -- 172.21.15.16:0/3159388915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa99406d360 msgr2=0x7fa99406deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.538+0000 7fa99ab70640 1 --2- 172.21.15.16:0/3159388915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa99406d360 0x7fa99406deb0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa98c009d90 tx=0x7fa98c02a680 comp rx=0 tx=0).stop 2024-04-24T23:49:41.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.538+0000 7fa99ab70640 1 -- 172.21.15.16:0/3159388915 shutdown_connections 2024-04-24T23:49:41.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.538+0000 7fa99ab70640 1 --2- 172.21.15.16:0/3159388915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa99406d360 0x7fa99406deb0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.538+0000 7fa99ab70640 1 -- 172.21.15.16:0/3159388915 >> 172.21.15.16:0/3159388915 conn(0x7fa9940affc0 msgr2=0x7fa9940b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:41.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.538+0000 7fa99ab70640 1 -- 172.21.15.16:0/3159388915 shutdown_connections 2024-04-24T23:49:41.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.538+0000 7fa99ab70640 1 -- 172.21.15.16:0/3159388915 wait complete. 2024-04-24T23:49:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.539+0000 7fa99ab70640 1 Processor -- start 2024-04-24T23:49:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.539+0000 7fa99ab70640 1 -- start start 2024-04-24T23:49:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.540+0000 7fa99ab70640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa99406d360 0x7fa994160db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.540+0000 7fa99ab70640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9940af980 0x7fa994150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.540+0000 7fa99ab70640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa99415dff0 0x7fa994151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.540+0000 7fa99ab70640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa9940ccca0 con 0x7fa9940af980 2024-04-24T23:49:41.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.540+0000 7fa99ab70640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa9940718f0 con 0x7fa99406d360 2024-04-24T23:49:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.540+0000 7fa99ab70640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fa9940cce20 con 0x7fa99415dff0 2024-04-24T23:49:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.540+0000 7fa993fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9940af980 0x7fa994150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.540+0000 7fa993fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9940af980 0x7fa994150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59744/0 (socket says 172.21.15.16:59744) 2024-04-24T23:49:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.540+0000 7fa993fff640 1 -- 172.21.15.16:0/1602737704 learned_addr learned my addr 172.21.15.16:0/1602737704 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.540+0000 7fa99890c640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa99406d360 0x7fa994160db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.541+0000 7fa99910d640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa99415dff0 0x7fa994151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.541+0000 7fa993fff640 1 -- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa99415dff0 msgr2=0x7fa994151e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.541+0000 7fa993fff640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa99415dff0 0x7fa994151e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.541+0000 7fa993fff640 1 -- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa99406d360 msgr2=0x7fa994160db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.541+0000 7fa993fff640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa99406d360 0x7fa994160db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.541+0000 7fa993fff640 1 -- 172.21.15.16:0/1602737704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa994153d10 con 0x7fa9940af980 2024-04-24T23:49:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.541+0000 7fa99910d640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa99415dff0 0x7fa994151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.541+0000 7fa993fff640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9940af980 0x7fa994150f40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa98000c780 tx=0x7fa98000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:41.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.541+0000 7fa9917fa640 1 -- 172.21.15.16:0/1602737704 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa980017070 con 0x7fa9940af980 2024-04-24T23:49:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.541+0000 7fa9917fa640 1 -- 172.21.15.16:0/1602737704 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa980005ad0 con 0x7fa9940af980 2024-04-24T23:49:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.541+0000 7fa99ab70640 1 -- 172.21.15.16:0/1602737704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9941612f0 con 0x7fa9940af980 2024-04-24T23:49:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.541+0000 7fa9917fa640 1 -- 172.21.15.16:0/1602737704 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa980007070 con 0x7fa9940af980 2024-04-24T23:49:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.541+0000 7fa99ab70640 1 -- 172.21.15.16:0/1602737704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa994153fb0 con 0x7fa9940af980 2024-04-24T23:49:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.542+0000 7fa99ab70640 1 -- 172.21.15.16:0/1602737704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa994068f90 con 0x7fa9940af980 2024-04-24T23:49:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.543+0000 7fa9917fa640 1 -- 172.21.15.16:0/1602737704 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa98001a070 con 0x7fa9940af980 2024-04-24T23:49:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.543+0000 7fa9917fa640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa97806be30 0x7fa97806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.543+0000 7fa9917fa640 1 -- 172.21.15.16:0/1602737704 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa98008a750 con 0x7fa9940af980 2024-04-24T23:49:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.543+0000 7fa99890c640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa97806be30 0x7fa97806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.544+0000 7fa99890c640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa97806be30 0x7fa97806e2f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa988000f60 tx=0x7fa988009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:41.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.547+0000 7fa9917fa640 1 -- 172.21.15.16:0/1602737704 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa980058630 con 0x7fa9940af980 2024-04-24T23:49:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.732+0000 7fa99ab70640 1 -- 172.21.15.16:0/1602737704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa994161730 con 0x7fa9940af980 2024-04-24T23:49:41.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.733+0000 7fa9917fa640 1 -- 172.21.15.16:0/1602737704 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fa98005c930 con 0x7fa9940af980 2024-04-24T23:49:41.732 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:41.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.734+0000 7fa99ab70640 1 -- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa97806be30 msgr2=0x7fa97806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.734+0000 7fa99ab70640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa97806be30 0x7fa97806e2f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa988000f60 tx=0x7fa988009040 comp rx=0 tx=0).stop 2024-04-24T23:49:41.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.734+0000 7fa99ab70640 1 -- 172.21.15.16:0/1602737704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9940af980 msgr2=0x7fa994150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.734+0000 7fa99ab70640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9940af980 0x7fa994150f40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa98000c780 tx=0x7fa98000cc40 comp rx=0 tx=0).stop 2024-04-24T23:49:41.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.735+0000 7fa99ab70640 1 -- 172.21.15.16:0/1602737704 shutdown_connections 2024-04-24T23:49:41.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.735+0000 7fa99ab70640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa97806be30 0x7fa97806e2f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.735+0000 7fa99ab70640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa99415dff0 0x7fa994151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.735+0000 7fa99ab70640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9940af980 0x7fa994150f40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.735+0000 7fa99ab70640 1 --2- 172.21.15.16:0/1602737704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa99406d360 0x7fa994160db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.735+0000 7fa99ab70640 1 -- 172.21.15.16:0/1602737704 >> 172.21.15.16:0/1602737704 conn(0x7fa9940affc0 msgr2=0x7fa99415aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:41.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.735+0000 7fa99ab70640 1 -- 172.21.15.16:0/1602737704 shutdown_connections 2024-04-24T23:49:41.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.735+0000 7fa99ab70640 1 -- 172.21.15.16:0/1602737704 wait complete. 2024-04-24T23:49:41.745 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:41.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.850+0000 7f2de9a7d640 1 Processor -- start 2024-04-24T23:49:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.850+0000 7f2de9a7d640 1 -- start start 2024-04-24T23:49:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.851+0000 7f2de9a7d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2de406bc40 0x7f2de406c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.851+0000 7f2de9a7d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2de40b7050 0x7f2de40b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.851+0000 7f2de9a7d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2de40d24d0 con 0x7f2de415dfd0 2024-04-24T23:49:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.851+0000 7f2de9a7d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f2de406c5e0 con 0x7f2de40b7050 2024-04-24T23:49:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.851+0000 7f2de9a7d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2de406ba00 con 0x7f2de406bc40 2024-04-24T23:49:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.851+0000 7f2de37fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2de415dfd0 0x7f2de406b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52288/0 (socket says 172.21.15.16:52288) 2024-04-24T23:49:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.851+0000 7f2de37fe640 1 -- 172.21.15.16:0/3471488561 learned_addr learned my addr 172.21.15.16:0/3471488561 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.851+0000 7f2de3fff640 1 --2- 172.21.15.16:0/3471488561 >> v2:172.21.15.156:3300/0 conn(0x7f2de40b7050 0x7f2de40b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.851+0000 7f2ddaffd640 1 --2- 172.21.15.16:0/3471488561 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2de406bc40 0x7f2de406c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.852+0000 7f2de2ffd640 1 -- 172.21.15.16:0/3471488561 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4227955186 0 0) 0x7f2de40d24d0 con 0x7f2de415dfd0 2024-04-24T23:49:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.852+0000 7f2de2ffd640 1 -- 172.21.15.16:0/3471488561 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2dd0003520 con 0x7f2de415dfd0 2024-04-24T23:49:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.852+0000 7f2de3fff640 1 -- 172.21.15.16:0/3471488561 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2de406bc40 msgr2=0x7f2de406c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.852+0000 7f2de3fff640 1 --2- 172.21.15.16:0/3471488561 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2de406bc40 0x7f2de406c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.852+0000 7f2de3fff640 1 -- 172.21.15.16:0/3471488561 >> v1:172.21.15.16:6789/0 conn(0x7f2de415dfd0 legacy=0x7f2de406b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.852+0000 7f2de3fff640 1 -- 172.21.15.16:0/3471488561 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de40d27b0 con 0x7f2de40b7050 2024-04-24T23:49:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.852+0000 7f2ddaffd640 1 --2- 172.21.15.16:0/3471488561 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2de406bc40 0x7f2de406c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:41.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.853+0000 7f2de3fff640 1 --2- 172.21.15.16:0/3471488561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de40b7050 0x7f2de40b74a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc009d90 tx=0x7f2ddc02a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ce239e41d060f6d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.853+0000 7f2de2ffd640 1 -- 172.21.15.16:0/3471488561 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc031600 con 0x7f2de40b7050 2024-04-24T23:49:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.853+0000 7f2de2ffd640 1 -- 172.21.15.16:0/3471488561 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ddc0317a0 con 0x7f2de40b7050 2024-04-24T23:49:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.853+0000 7f2de2ffd640 1 -- 172.21.15.16:0/3471488561 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc031b70 con 0x7f2de40b7050 2024-04-24T23:49:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.853+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3471488561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de40b7050 msgr2=0x7f2de40b74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.853+0000 7f2de9a7d640 1 --2- 172.21.15.16:0/3471488561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de40b7050 0x7f2de40b74a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc009d90 tx=0x7f2ddc02a5d0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.853+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3471488561 shutdown_connections 2024-04-24T23:49:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.853+0000 7f2de9a7d640 1 --2- 172.21.15.16:0/3471488561 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de40b7050 0x7f2de40b74a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.853+0000 7f2de9a7d640 1 --2- 172.21.15.16:0/3471488561 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2de406bc40 0x7f2de406c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.853+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3471488561 >> 172.21.15.16:0/3471488561 conn(0x7f2de40d3dc0 msgr2=0x7f2de40d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.854+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3471488561 shutdown_connections 2024-04-24T23:49:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.854+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3471488561 wait complete. 2024-04-24T23:49:41.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.855+0000 7f2de9a7d640 1 Processor -- start 2024-04-24T23:49:41.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.855+0000 7f2de9a7d640 1 -- start start 2024-04-24T23:49:41.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2de9a7d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2de406bc40 0x7f2de41e92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2de9a7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de40b7050 0x7f2de41e8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2de9a7d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de415dfd0 0x7f2de41e66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2de9a7d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2de40d3050 con 0x7f2de40b7050 2024-04-24T23:49:41.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2de9a7d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2de40d2d70 con 0x7f2de415dfd0 2024-04-24T23:49:41.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2de9a7d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2de40cead0 con 0x7f2de406bc40 2024-04-24T23:49:41.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2ddaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de40b7050 0x7f2de41e8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2ddaffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de40b7050 0x7f2de41e8230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59748/0 (socket says 172.21.15.16:59748) 2024-04-24T23:49:41.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2ddaffd640 1 -- 172.21.15.16:0/3453918155 learned_addr learned my addr 172.21.15.16:0/3453918155 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:41.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2de3fff640 1 --2- 172.21.15.16:0/3453918155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de415dfd0 0x7f2de41e66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2de37fe640 1 --2- 172.21.15.16:0/3453918155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2de406bc40 0x7f2de41e92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2ddaffd640 1 -- 172.21.15.16:0/3453918155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2de406bc40 msgr2=0x7f2de41e92f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2ddaffd640 1 --2- 172.21.15.16:0/3453918155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2de406bc40 0x7f2de41e92f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2ddaffd640 1 -- 172.21.15.16:0/3453918155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de415dfd0 msgr2=0x7f2de41e66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:41.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2ddaffd640 1 --2- 172.21.15.16:0/3453918155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de415dfd0 0x7f2de41e66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:41.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2ddaffd640 1 -- 172.21.15.16:0/3453918155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2de41e7190 con 0x7f2de40b7050 2024-04-24T23:49:41.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.856+0000 7f2ddaffd640 1 --2- 172.21.15.16:0/3453918155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de40b7050 0x7f2de41e8230 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00ebe0 tx=0x7f2dcc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:41.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.857+0000 7f2de0ff9640 1 -- 172.21.15.16:0/3453918155 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc017070 con 0x7f2de40b7050 2024-04-24T23:49:41.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.857+0000 7f2de0ff9640 1 -- 172.21.15.16:0/3453918155 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2dcc00cc30 con 0x7f2de40b7050 2024-04-24T23:49:41.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.857+0000 7f2de0ff9640 1 -- 172.21.15.16:0/3453918155 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2dcc013470 con 0x7f2de40b7050 2024-04-24T23:49:41.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.857+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3453918155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2de41e7470 con 0x7f2de40b7050 2024-04-24T23:49:41.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.857+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3453918155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2de41e7a30 con 0x7f2de40b7050 2024-04-24T23:49:41.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.858+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3453918155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2de4068f90 con 0x7f2de40b7050 2024-04-24T23:49:41.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.858+0000 7f2de0ff9640 1 -- 172.21.15.16:0/3453918155 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2dcc01a070 con 0x7f2de40b7050 2024-04-24T23:49:41.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.859+0000 7f2de0ff9640 1 --2- 172.21.15.16:0/3453918155 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2dc406be30 0x7f2dc406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:41.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.859+0000 7f2de0ff9640 1 -- 172.21.15.16:0/3453918155 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f2dcc08b4f0 con 0x7f2de40b7050 2024-04-24T23:49:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.859+0000 7f2de37fe640 1 --2- 172.21.15.16:0/3453918155 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2dc406be30 0x7f2dc406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.860+0000 7f2de37fe640 1 --2- 172.21.15.16:0/3453918155 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2dc406be30 0x7f2dc406e2f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4005880 tx=0x7f2dd4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:41.863+0000 7f2de0ff9640 1 -- 172.21.15.16:0/3453918155 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2dcc059b30 con 0x7f2de40b7050 2024-04-24T23:49:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.047+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3453918155 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2de415ce90 con 0x7f2de40b7050 2024-04-24T23:49:42.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.047+0000 7f2de0ff9640 1 -- 172.21.15.16:0/3453918155 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f2dcc059430 con 0x7f2de40b7050 2024-04-24T23:49:42.047 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.049+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3453918155 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2dc406be30 msgr2=0x7f2dc406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.049+0000 7f2de9a7d640 1 --2- 172.21.15.16:0/3453918155 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2dc406be30 0x7f2dc406e2f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4005880 tx=0x7f2dd4009040 comp rx=0 tx=0).stop 2024-04-24T23:49:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.049+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3453918155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de40b7050 msgr2=0x7f2de41e8230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.049+0000 7f2de9a7d640 1 --2- 172.21.15.16:0/3453918155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de40b7050 0x7f2de41e8230 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2dcc00ebe0 tx=0x7f2dcc00c520 comp rx=0 tx=0).stop 2024-04-24T23:49:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.049+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3453918155 shutdown_connections 2024-04-24T23:49:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.049+0000 7f2de9a7d640 1 --2- 172.21.15.16:0/3453918155 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2dc406be30 0x7f2dc406e2f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.049+0000 7f2de9a7d640 1 --2- 172.21.15.16:0/3453918155 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2de415dfd0 0x7f2de41e66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.049+0000 7f2de9a7d640 1 --2- 172.21.15.16:0/3453918155 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2de40b7050 0x7f2de41e8230 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.049+0000 7f2de9a7d640 1 --2- 172.21.15.16:0/3453918155 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2de406bc40 0x7f2de41e92f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.049+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3453918155 >> 172.21.15.16:0/3453918155 conn(0x7f2de40d3dc0 msgr2=0x7f2de406b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.049+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3453918155 shutdown_connections 2024-04-24T23:49:42.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.049+0000 7f2de9a7d640 1 -- 172.21.15.16:0/3453918155 wait complete. 2024-04-24T23:49:42.060 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:42.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.165+0000 7f61d068c640 1 Processor -- start 2024-04-24T23:49:42.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.165+0000 7f61d068c640 1 -- start start 2024-04-24T23:49:42.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.166+0000 7f61d068c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f61c81444c0 0x7f61c8071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.166+0000 7f61d068c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f61c8071fd0 0x7f61c80ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.166+0000 7f61d068c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61c80cf0a0 con 0x7f61c80ce870 2024-04-24T23:49:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.166+0000 7f61d068c640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f61c8072430 con 0x7f61c81444c0 2024-04-24T23:49:42.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.166+0000 7f61d068c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f61c806bbf0 con 0x7f61c8071fd0 2024-04-24T23:49:42.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.166+0000 7f61ce428640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f61c81444c0 0x7f61c8071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.166+0000 7f61cec29640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f61c80ce870 0x7f61c80b3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52304/0 (socket says 172.21.15.16:52304) 2024-04-24T23:49:42.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.166+0000 7f61cec29640 1 -- 172.21.15.16:0/3851478634 learned_addr learned my addr 172.21.15.16:0/3851478634 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:42.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.166+0000 7f61ce428640 1 --2- 172.21.15.16:0/3851478634 >> v2:172.21.15.16:3300/0 conn(0x7f61c81444c0 0x7f61c8071a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59760/0 (socket says 172.21.15.16:59760) 2024-04-24T23:49:42.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.166+0000 7f61cdc27640 1 --2- 172.21.15.16:0/3851478634 >> v2:172.21.15.156:3300/0 conn(0x7f61c8071fd0 0x7f61c80ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.166+0000 7f61ce428640 1 -- 172.21.15.16:0/3851478634 >> v2:172.21.15.156:3300/0 conn(0x7f61c8071fd0 msgr2=0x7f61c80ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:42.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.166+0000 7f61ce428640 1 --2- 172.21.15.16:0/3851478634 >> v2:172.21.15.156:3300/0 conn(0x7f61c8071fd0 0x7f61c80ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.166+0000 7f61ce428640 1 -- 172.21.15.16:0/3851478634 >> v1:172.21.15.16:6789/0 conn(0x7f61c80ce870 legacy=0x7f61c80b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.166+0000 7f61ce428640 1 -- 172.21.15.16:0/3851478634 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c806be90 con 0x7f61c81444c0 2024-04-24T23:49:42.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.167+0000 7f61ce428640 1 --2- 172.21.15.16:0/3851478634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c81444c0 0x7f61c8071a90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f61c806ec30 tx=0x7f61c402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84784583d68033d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:42.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.167+0000 7f61cd426640 1 -- 172.21.15.16:0/3851478634 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c4037070 con 0x7f61c81444c0 2024-04-24T23:49:42.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.167+0000 7f61cd426640 1 -- 172.21.15.16:0/3851478634 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61c4032440 con 0x7f61c81444c0 2024-04-24T23:49:42.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.167+0000 7f61cd426640 1 -- 172.21.15.16:0/3851478634 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c4032810 con 0x7f61c81444c0 2024-04-24T23:49:42.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.168+0000 7f61d068c640 1 -- 172.21.15.16:0/3851478634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c81444c0 msgr2=0x7f61c8071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.168+0000 7f61d068c640 1 --2- 172.21.15.16:0/3851478634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c81444c0 0x7f61c8071a90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f61c806ec30 tx=0x7f61c402a1a0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.168+0000 7f61d068c640 1 -- 172.21.15.16:0/3851478634 shutdown_connections 2024-04-24T23:49:42.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.168+0000 7f61d068c640 1 --2- 172.21.15.16:0/3851478634 >> v2:172.21.15.156:3300/0 conn(0x7f61c8071fd0 0x7f61c80ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.168+0000 7f61d068c640 1 --2- 172.21.15.16:0/3851478634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c81444c0 0x7f61c8071a90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.168+0000 7f61d068c640 1 -- 172.21.15.16:0/3851478634 >> 172.21.15.16:0/3851478634 conn(0x7f61c80cfc50 msgr2=0x7f61c80d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:42.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.168+0000 7f61d068c640 1 -- 172.21.15.16:0/3851478634 shutdown_connections 2024-04-24T23:49:42.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.168+0000 7f61d068c640 1 -- 172.21.15.16:0/3851478634 wait complete. 2024-04-24T23:49:42.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.169+0000 7f61d068c640 1 Processor -- start 2024-04-24T23:49:42.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.169+0000 7f61d068c640 1 -- start start 2024-04-24T23:49:42.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61d068c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c8071fd0 0x7f61c81538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61d068c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61c80ce870 0x7f61c81591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61d068c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61c81444c0 0x7f61c815d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61d068c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f61c806c730 con 0x7f61c8071fd0 2024-04-24T23:49:42.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61d068c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f61c806c1d0 con 0x7f61c80ce870 2024-04-24T23:49:42.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61d068c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f61c80cecc0 con 0x7f61c81444c0 2024-04-24T23:49:42.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61ce428640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c8071fd0 0x7f61c81538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61ce428640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c8071fd0 0x7f61c81538e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59768/0 (socket says 172.21.15.16:59768) 2024-04-24T23:49:42.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61ce428640 1 -- 172.21.15.16:0/174314046 learned_addr learned my addr 172.21.15.16:0/174314046 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:42.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61cdc27640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61c80ce870 0x7f61c81591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61cec29640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61c81444c0 0x7f61c815d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61ce428640 1 -- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61c81444c0 msgr2=0x7f61c815d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61ce428640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61c81444c0 0x7f61c815d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61ce428640 1 -- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61c80ce870 msgr2=0x7f61c81591a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61ce428640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61c80ce870 0x7f61c81591a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61ce428640 1 -- 172.21.15.16:0/174314046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61c815a960 con 0x7f61c8071fd0 2024-04-24T23:49:42.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61cdc27640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61c80ce870 0x7f61c81591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:42.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61cec29640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61c81444c0 0x7f61c815d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:42.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61ce428640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c8071fd0 0x7f61c81538e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f61c4000c00 tx=0x7f61c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:42.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61beffd640 1 -- 172.21.15.16:0/174314046 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c4037040 con 0x7f61c8071fd0 2024-04-24T23:49:42.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61beffd640 1 -- 172.21.15.16:0/174314046 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61c4042070 con 0x7f61c8071fd0 2024-04-24T23:49:42.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61d068c640 1 -- 172.21.15.16:0/174314046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61c815ac40 con 0x7f61c8071fd0 2024-04-24T23:49:42.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61beffd640 1 -- 172.21.15.16:0/174314046 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c4030bd0 con 0x7f61c8071fd0 2024-04-24T23:49:42.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.170+0000 7f61d068c640 1 -- 172.21.15.16:0/174314046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61c81596e0 con 0x7f61c8071fd0 2024-04-24T23:49:42.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.171+0000 7f61d068c640 1 -- 172.21.15.16:0/174314046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61c8068f90 con 0x7f61c8071fd0 2024-04-24T23:49:42.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.172+0000 7f61beffd640 1 -- 172.21.15.16:0/174314046 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f61c4047460 con 0x7f61c8071fd0 2024-04-24T23:49:42.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.172+0000 7f61beffd640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f61ac06be30 0x7f61ac06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.172+0000 7f61beffd640 1 -- 172.21.15.16:0/174314046 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f61c40abe20 con 0x7f61c8071fd0 2024-04-24T23:49:42.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.172+0000 7f61cdc27640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f61ac06be30 0x7f61ac06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.173+0000 7f61cdc27640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f61ac06be30 0x7f61ac06e2f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f61b8006fd0 tx=0x7f61b8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:42.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.176+0000 7f61beffd640 1 -- 172.21.15.16:0/174314046 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f61c4032e20 con 0x7f61c8071fd0 2024-04-24T23:49:42.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.361+0000 7f61d068c640 1 -- 172.21.15.16:0/174314046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f61c815b000 con 0x7f61c8071fd0 2024-04-24T23:49:42.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.361+0000 7f61beffd640 1 -- 172.21.15.16:0/174314046 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f61c407a460 con 0x7f61c8071fd0 2024-04-24T23:49:42.361 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:42.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.363+0000 7f61d068c640 1 -- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f61ac06be30 msgr2=0x7f61ac06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.363+0000 7f61d068c640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f61ac06be30 0x7f61ac06e2f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f61b8006fd0 tx=0x7f61b8008040 comp rx=0 tx=0).stop 2024-04-24T23:49:42.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.363+0000 7f61d068c640 1 -- 172.21.15.16:0/174314046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c8071fd0 msgr2=0x7f61c81538e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.363+0000 7f61d068c640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c8071fd0 0x7f61c81538e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f61c4000c00 tx=0x7f61c40046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.363+0000 7f61d068c640 1 -- 172.21.15.16:0/174314046 shutdown_connections 2024-04-24T23:49:42.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.363+0000 7f61d068c640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f61ac06be30 0x7f61ac06e2f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.363+0000 7f61d068c640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61c81444c0 0x7f61c815d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.363+0000 7f61d068c640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61c80ce870 0x7f61c81591a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.363+0000 7f61d068c640 1 --2- 172.21.15.16:0/174314046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61c8071fd0 0x7f61c81538e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.363+0000 7f61d068c640 1 -- 172.21.15.16:0/174314046 >> 172.21.15.16:0/174314046 conn(0x7f61c80cfc50 msgr2=0x7f61c80b0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:42.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.363+0000 7f61d068c640 1 -- 172.21.15.16:0/174314046 shutdown_connections 2024-04-24T23:49:42.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.363+0000 7f61d068c640 1 -- 172.21.15.16:0/174314046 wait complete. 2024-04-24T23:49:42.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:42.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.475+0000 7f1af3c6b640 1 Processor -- start 2024-04-24T23:49:42.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.475+0000 7f1af3c6b640 1 -- start start 2024-04-24T23:49:42.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af3c6b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1aec15dfa0 0x7f1aec0d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af3c6b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1aec0ce5f0 0x7f1aec0cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af3c6b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1aec0b5f00 0x7f1aec0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af3c6b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1aec0b7360 con 0x7f1aec0ce5f0 2024-04-24T23:49:42.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af3c6b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f1aec0cc640 con 0x7f1aec15dfa0 2024-04-24T23:49:42.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af3c6b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1aec0d05a0 con 0x7f1aec0b5f00 2024-04-24T23:49:42.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af1206640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1aec0ce5f0 0x7f1aec0cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af1206640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1aec0ce5f0 0x7f1aec0cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59784/0 (socket says 172.21.15.16:59784) 2024-04-24T23:49:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af1206640 1 -- 172.21.15.16:0/1677267711 learned_addr learned my addr 172.21.15.16:0/1677267711 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af1a07640 1 --2- 172.21.15.16:0/1677267711 >> v2:172.21.15.156:3300/0 conn(0x7f1aec15dfa0 0x7f1aec0d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af2208640 1 --2- 172.21.15.16:0/1677267711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1aec0b5f00 0x7f1aec0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af1206640 1 -- 172.21.15.16:0/1677267711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1aec0b5f00 msgr2=0x7f1aec0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af1206640 1 --2- 172.21.15.16:0/1677267711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1aec0b5f00 0x7f1aec0b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af1206640 1 -- 172.21.15.16:0/1677267711 >> v2:172.21.15.156:3300/0 conn(0x7f1aec15dfa0 msgr2=0x7f1aec0d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af1206640 1 --2- 172.21.15.16:0/1677267711 >> v2:172.21.15.156:3300/0 conn(0x7f1aec15dfa0 0x7f1aec0d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.476+0000 7f1af1206640 1 -- 172.21.15.16:0/1677267711 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aec0d0880 con 0x7f1aec0ce5f0 2024-04-24T23:49:42.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.477+0000 7f1af1206640 1 --2- 172.21.15.16:0/1677267711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aec0ce5f0 0x7f1aec0cea70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1aec0cf3e0 tx=0x7f1adc02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=104d478841f1d955 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.477+0000 7f1af0a05640 1 -- 172.21.15.16:0/1677267711 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1adc037070 con 0x7f1aec0ce5f0 2024-04-24T23:49:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.477+0000 7f1af0a05640 1 -- 172.21.15.16:0/1677267711 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1adc032440 con 0x7f1aec0ce5f0 2024-04-24T23:49:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.477+0000 7f1af0a05640 1 -- 172.21.15.16:0/1677267711 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1adc032810 con 0x7f1aec0ce5f0 2024-04-24T23:49:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.478+0000 7f1af3c6b640 1 -- 172.21.15.16:0/1677267711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aec0ce5f0 msgr2=0x7f1aec0cea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.478+0000 7f1af3c6b640 1 --2- 172.21.15.16:0/1677267711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aec0ce5f0 0x7f1aec0cea70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1aec0cf3e0 tx=0x7f1adc02a1e0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.478+0000 7f1af3c6b640 1 -- 172.21.15.16:0/1677267711 shutdown_connections 2024-04-24T23:49:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.478+0000 7f1af3c6b640 1 --2- 172.21.15.16:0/1677267711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1aec0b5f00 0x7f1aec0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.478+0000 7f1af3c6b640 1 --2- 172.21.15.16:0/1677267711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aec0ce5f0 0x7f1aec0cea70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.478+0000 7f1af3c6b640 1 --2- 172.21.15.16:0/1677267711 >> v2:172.21.15.156:3300/0 conn(0x7f1aec15dfa0 0x7f1aec0d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.478+0000 7f1af3c6b640 1 -- 172.21.15.16:0/1677267711 >> 172.21.15.16:0/1677267711 conn(0x7f1aec06b4d0 msgr2=0x7f1aec0b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.478+0000 7f1af3c6b640 1 -- 172.21.15.16:0/1677267711 shutdown_connections 2024-04-24T23:49:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.478+0000 7f1af3c6b640 1 -- 172.21.15.16:0/1677267711 wait complete. 2024-04-24T23:49:42.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.479+0000 7f1af3c6b640 1 Processor -- start 2024-04-24T23:49:42.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.479+0000 7f1af3c6b640 1 -- start start 2024-04-24T23:49:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af3c6b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1aec0b5f00 0x7f1aec1470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af3c6b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1aec0ce5f0 0x7f1aec0c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af3c6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aec15dfa0 0x7f1aec0c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af3c6b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec15e380 con 0x7f1aec15dfa0 2024-04-24T23:49:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af3c6b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1aec0cc2d0 con 0x7f1aec0b5f00 2024-04-24T23:49:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af3c6b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1aec0d1520 con 0x7f1aec0ce5f0 2024-04-24T23:49:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af2208640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aec15dfa0 0x7f1aec0c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af2208640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aec15dfa0 0x7f1aec0c74c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59798/0 (socket says 172.21.15.16:59798) 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af2208640 1 -- 172.21.15.16:0/3433840024 learned_addr learned my addr 172.21.15.16:0/3433840024 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af1a07640 1 --2- 172.21.15.16:0/3433840024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1aec0b5f00 0x7f1aec1470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af1206640 1 --2- 172.21.15.16:0/3433840024 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1aec0ce5f0 0x7f1aec0c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af2208640 1 -- 172.21.15.16:0/3433840024 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1aec0ce5f0 msgr2=0x7f1aec0c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af2208640 1 --2- 172.21.15.16:0/3433840024 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1aec0ce5f0 0x7f1aec0c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af2208640 1 -- 172.21.15.16:0/3433840024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1aec0b5f00 msgr2=0x7f1aec1470f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af2208640 1 --2- 172.21.15.16:0/3433840024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1aec0b5f00 0x7f1aec1470f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af2208640 1 -- 172.21.15.16:0/3433840024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aec0c5cc0 con 0x7f1aec15dfa0 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.480+0000 7f1af2208640 1 --2- 172.21.15.16:0/3433840024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aec15dfa0 0x7f1aec0c74c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1ae400eed0 tx=0x7f1ae400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.481+0000 7f1ae27fc640 1 -- 172.21.15.16:0/3433840024 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae4018020 con 0x7f1aec15dfa0 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.481+0000 7f1ae27fc640 1 -- 172.21.15.16:0/3433840024 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ae4014400 con 0x7f1aec15dfa0 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.481+0000 7f1af3c6b640 1 -- 172.21.15.16:0/3433840024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aec0c5fa0 con 0x7f1aec15dfa0 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.481+0000 7f1ae27fc640 1 -- 172.21.15.16:0/3433840024 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ae40147f0 con 0x7f1aec15dfa0 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.481+0000 7f1af3c6b640 1 -- 172.21.15.16:0/3433840024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aec1534e0 con 0x7f1aec15dfa0 2024-04-24T23:49:42.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.482+0000 7f1af3c6b640 1 -- 172.21.15.16:0/3433840024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aec06bea0 con 0x7f1aec15dfa0 2024-04-24T23:49:42.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.482+0000 7f1ae27fc640 1 -- 172.21.15.16:0/3433840024 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1ae4005d10 con 0x7f1aec15dfa0 2024-04-24T23:49:42.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.482+0000 7f1ae27fc640 1 --2- 172.21.15.16:0/3433840024 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1ad006be60 0x7f1ad006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.482+0000 7f1ae27fc640 1 -- 172.21.15.16:0/3433840024 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1ae408b8a0 con 0x7f1aec15dfa0 2024-04-24T23:49:42.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.483+0000 7f1af1a07640 1 --2- 172.21.15.16:0/3433840024 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1ad006be60 0x7f1ad006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.483+0000 7f1af1a07640 1 --2- 172.21.15.16:0/3433840024 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1ad006be60 0x7f1ad006e320 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8009a30 tx=0x7f1ae8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:42.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.487+0000 7f1ae27fc640 1 -- 172.21.15.16:0/3433840024 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1ae4022070 con 0x7f1aec15dfa0 2024-04-24T23:49:42.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.672+0000 7f1af3c6b640 1 -- 172.21.15.16:0/3433840024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1aec04f5e0 con 0x7f1aec15dfa0 2024-04-24T23:49:42.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.673+0000 7f1ae27fc640 1 -- 172.21.15.16:0/3433840024 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7f1ae4059ee0 con 0x7f1aec15dfa0 2024-04-24T23:49:42.673 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:42.673 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.675+0000 7f1af3c6b640 1 -- 172.21.15.16:0/3433840024 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1ad006be60 msgr2=0x7f1ad006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.675+0000 7f1af3c6b640 1 --2- 172.21.15.16:0/3433840024 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1ad006be60 0x7f1ad006e320 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8009a30 tx=0x7f1ae8008040 comp rx=0 tx=0).stop 2024-04-24T23:49:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.675+0000 7f1af3c6b640 1 -- 172.21.15.16:0/3433840024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aec15dfa0 msgr2=0x7f1aec0c74c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.675+0000 7f1af3c6b640 1 --2- 172.21.15.16:0/3433840024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aec15dfa0 0x7f1aec0c74c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1ae400eed0 tx=0x7f1ae400b480 comp rx=0 tx=0).stop 2024-04-24T23:49:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.675+0000 7f1af3c6b640 1 -- 172.21.15.16:0/3433840024 shutdown_connections 2024-04-24T23:49:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.675+0000 7f1af3c6b640 1 --2- 172.21.15.16:0/3433840024 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1ad006be60 0x7f1ad006e320 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.675+0000 7f1af3c6b640 1 --2- 172.21.15.16:0/3433840024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1aec15dfa0 0x7f1aec0c74c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.675+0000 7f1af3c6b640 1 --2- 172.21.15.16:0/3433840024 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1aec0ce5f0 0x7f1aec0c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.675+0000 7f1af3c6b640 1 --2- 172.21.15.16:0/3433840024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1aec0b5f00 0x7f1aec1470f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.675+0000 7f1af3c6b640 1 -- 172.21.15.16:0/3433840024 >> 172.21.15.16:0/3433840024 conn(0x7f1aec06b4d0 msgr2=0x7f1aec160860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.675+0000 7f1af3c6b640 1 -- 172.21.15.16:0/3433840024 shutdown_connections 2024-04-24T23:49:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.675+0000 7f1af3c6b640 1 -- 172.21.15.16:0/3433840024 wait complete. 2024-04-24T23:49:42.686 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:42.686 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:42.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.790+0000 7f330fa22640 1 Processor -- start 2024-04-24T23:49:42.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.790+0000 7f330fa22640 1 -- start start 2024-04-24T23:49:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.791+0000 7f330fa22640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f33080af980 0x7f330806deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.791+0000 7f330fa22640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330806d360 0x7f3308071e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.791+0000 7f330fa22640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f330806d8d0 con 0x7f330815dff0 2024-04-24T23:49:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.791+0000 7f330fa22640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f33080732d0 con 0x7f33080af980 2024-04-24T23:49:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.791+0000 7f330fa22640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f330806dbb0 con 0x7f330806d360 2024-04-24T23:49:42.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.791+0000 7f330d7be640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f330815dff0 0x7f33080af270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52312/0 (socket says 172.21.15.16:52312) 2024-04-24T23:49:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.791+0000 7f330cfbd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f33080af980 0x7f330806deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.791+0000 7f330cfbd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f33080af980 0x7f330806deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59800/0 (socket says 172.21.15.16:59800) 2024-04-24T23:49:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.791+0000 7f330d7be640 1 -- 172.21.15.16:0/4132785642 learned_addr learned my addr 172.21.15.16:0/4132785642 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.791+0000 7f330dfbf640 1 --2- 172.21.15.16:0/4132785642 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330806d360 0x7f3308071e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.792+0000 7f32fffff640 1 -- 172.21.15.16:0/4132785642 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2359658323 0 0) 0x7f330806d8d0 con 0x7f330815dff0 2024-04-24T23:49:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.792+0000 7f330cfbd640 1 -- 172.21.15.16:0/4132785642 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330806d360 msgr2=0x7f3308071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.792+0000 7f330cfbd640 1 --2- 172.21.15.16:0/4132785642 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330806d360 0x7f3308071e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.792+0000 7f330cfbd640 1 -- 172.21.15.16:0/4132785642 >> v1:172.21.15.16:6789/0 conn(0x7f330815dff0 legacy=0x7f33080af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.792+0000 7f330cfbd640 1 -- 172.21.15.16:0/4132785642 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3308071330 con 0x7f33080af980 2024-04-24T23:49:42.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.792+0000 7f330dfbf640 1 --2- 172.21.15.16:0/4132785642 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330806d360 0x7f3308071e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:42.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.792+0000 7f330cfbd640 1 --2- 172.21.15.16:0/4132785642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33080af980 0x7f330806deb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f32f80097a0 tx=0x7f32f802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a36b674d366bff server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:42.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.792+0000 7f32fffff640 1 -- 172.21.15.16:0/4132785642 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f8037070 con 0x7f33080af980 2024-04-24T23:49:42.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.793+0000 7f32fffff640 1 -- 172.21.15.16:0/4132785642 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32f8032440 con 0x7f33080af980 2024-04-24T23:49:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.793+0000 7f32fffff640 1 -- 172.21.15.16:0/4132785642 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32f8032810 con 0x7f33080af980 2024-04-24T23:49:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.793+0000 7f330fa22640 1 -- 172.21.15.16:0/4132785642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33080af980 msgr2=0x7f330806deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.793+0000 7f330fa22640 1 --2- 172.21.15.16:0/4132785642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33080af980 0x7f330806deb0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f32f80097a0 tx=0x7f32f802a180 comp rx=0 tx=0).stop 2024-04-24T23:49:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.793+0000 7f330fa22640 1 -- 172.21.15.16:0/4132785642 shutdown_connections 2024-04-24T23:49:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.793+0000 7f330fa22640 1 --2- 172.21.15.16:0/4132785642 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330806d360 0x7f3308071e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.793+0000 7f330fa22640 1 --2- 172.21.15.16:0/4132785642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33080af980 0x7f330806deb0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.793+0000 7f330fa22640 1 -- 172.21.15.16:0/4132785642 >> 172.21.15.16:0/4132785642 conn(0x7f33080affc0 msgr2=0x7f33080b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.793+0000 7f330fa22640 1 -- 172.21.15.16:0/4132785642 shutdown_connections 2024-04-24T23:49:42.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.793+0000 7f330fa22640 1 -- 172.21.15.16:0/4132785642 wait complete. 2024-04-24T23:49:42.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.794+0000 7f330fa22640 1 Processor -- start 2024-04-24T23:49:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.794+0000 7f330fa22640 1 -- start start 2024-04-24T23:49:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.795+0000 7f330fa22640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330806d360 0x7f3308145ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.795+0000 7f330fa22640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33080af980 0x7f33080ce050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.795+0000 7f330fa22640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330815dff0 0x7f33080ce590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.795+0000 7f330fa22640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f33080ccca0 con 0x7f330806d360 2024-04-24T23:49:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.795+0000 7f330fa22640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f33080718f0 con 0x7f33080af980 2024-04-24T23:49:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.795+0000 7f330fa22640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f33080cce20 con 0x7f330815dff0 2024-04-24T23:49:42.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.795+0000 7f330d7be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330806d360 0x7f3308145ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.795+0000 7f330d7be640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330806d360 0x7f3308145ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59802/0 (socket says 172.21.15.16:59802) 2024-04-24T23:49:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.795+0000 7f330d7be640 1 -- 172.21.15.16:0/1066724049 learned_addr learned my addr 172.21.15.16:0/1066724049 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.795+0000 7f330d7be640 1 -- 172.21.15.16:0/1066724049 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330815dff0 msgr2=0x7f33080ce590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.795+0000 7f330d7be640 1 --2- 172.21.15.16:0/1066724049 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330815dff0 0x7f33080ce590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.795+0000 7f330d7be640 1 -- 172.21.15.16:0/1066724049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33080af980 msgr2=0x7f33080ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.795+0000 7f330d7be640 1 --2- 172.21.15.16:0/1066724049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33080af980 0x7f33080ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.795+0000 7f330d7be640 1 -- 172.21.15.16:0/1066724049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3308155b00 con 0x7f330806d360 2024-04-24T23:49:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.796+0000 7f330d7be640 1 --2- 172.21.15.16:0/1066724049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330806d360 0x7f3308145ba0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f330400c430 tx=0x7f330400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.796+0000 7f32fdffb640 1 -- 172.21.15.16:0/1066724049 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33040074e0 con 0x7f330806d360 2024-04-24T23:49:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.796+0000 7f32fdffb640 1 -- 172.21.15.16:0/1066724049 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33040047e0 con 0x7f330806d360 2024-04-24T23:49:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.796+0000 7f32fdffb640 1 -- 172.21.15.16:0/1066724049 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3304004bd0 con 0x7f330806d360 2024-04-24T23:49:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.796+0000 7f330fa22640 1 -- 172.21.15.16:0/1066724049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33081460e0 con 0x7f330806d360 2024-04-24T23:49:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.796+0000 7f330fa22640 1 -- 172.21.15.16:0/1066724049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3308155da0 con 0x7f330806d360 2024-04-24T23:49:42.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.797+0000 7f330fa22640 1 -- 172.21.15.16:0/1066724049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3308068f90 con 0x7f330806d360 2024-04-24T23:49:42.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.797+0000 7f32fdffb640 1 -- 172.21.15.16:0/1066724049 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3304011070 con 0x7f330806d360 2024-04-24T23:49:42.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.798+0000 7f32fdffb640 1 --2- 172.21.15.16:0/1066724049 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f32ec06be60 0x7f32ec06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:42.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.798+0000 7f32fdffb640 1 -- 172.21.15.16:0/1066724049 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f330408ae90 con 0x7f330806d360 2024-04-24T23:49:42.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.798+0000 7f330cfbd640 1 --2- 172.21.15.16:0/1066724049 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f32ec06be60 0x7f32ec06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:42.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.798+0000 7f330cfbd640 1 --2- 172.21.15.16:0/1066724049 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f32ec06be60 0x7f32ec06e320 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f32f8034000 tx=0x7f32f802c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:42.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.802+0000 7f32fdffb640 1 -- 172.21.15.16:0/1066724049 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f330400e280 con 0x7f330806d360 2024-04-24T23:49:42.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.982+0000 7f330fa22640 1 -- 172.21.15.16:0/1066724049 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f33080ae260 con 0x7f330806d360 2024-04-24T23:49:42.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.983+0000 7f32fdffb640 1 -- 172.21.15.16:0/1066724049 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f3304059440 con 0x7f330806d360 2024-04-24T23:49:42.982 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.984+0000 7f330fa22640 1 -- 172.21.15.16:0/1066724049 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f32ec06be60 msgr2=0x7f32ec06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.985+0000 7f330fa22640 1 --2- 172.21.15.16:0/1066724049 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f32ec06be60 0x7f32ec06e320 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f32f8034000 tx=0x7f32f802c000 comp rx=0 tx=0).stop 2024-04-24T23:49:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.985+0000 7f330fa22640 1 -- 172.21.15.16:0/1066724049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330806d360 msgr2=0x7f3308145ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.985+0000 7f330fa22640 1 --2- 172.21.15.16:0/1066724049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330806d360 0x7f3308145ba0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f330400c430 tx=0x7f330400c8f0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.985+0000 7f330fa22640 1 -- 172.21.15.16:0/1066724049 shutdown_connections 2024-04-24T23:49:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.985+0000 7f330fa22640 1 --2- 172.21.15.16:0/1066724049 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f32ec06be60 0x7f32ec06e320 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.985+0000 7f330fa22640 1 --2- 172.21.15.16:0/1066724049 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f330815dff0 0x7f33080ce590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.985+0000 7f330fa22640 1 --2- 172.21.15.16:0/1066724049 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33080af980 0x7f33080ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.985+0000 7f330fa22640 1 --2- 172.21.15.16:0/1066724049 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f330806d360 0x7f3308145ba0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.985+0000 7f330fa22640 1 -- 172.21.15.16:0/1066724049 >> 172.21.15.16:0/1066724049 conn(0x7f33080affc0 msgr2=0x7f330806c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.985+0000 7f330fa22640 1 -- 172.21.15.16:0/1066724049 shutdown_connections 2024-04-24T23:49:42.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:42.985+0000 7f330fa22640 1 -- 172.21.15.16:0/1066724049 wait complete. 2024-04-24T23:49:42.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:43.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.099+0000 7feb1d314640 1 Processor -- start 2024-04-24T23:49:43.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.099+0000 7feb1d314640 1 -- start start 2024-04-24T23:49:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.100+0000 7feb1d314640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7feb1815d0a0 0x7feb180ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.100+0000 7feb1d314640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb180ceca0 0x7feb1806ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.100+0000 7feb1d314640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb1815f9c0 con 0x7feb181455f0 2024-04-24T23:49:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.100+0000 7feb1d314640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7feb180cf100 con 0x7feb180ceca0 2024-04-24T23:49:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.100+0000 7feb1d314640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7feb181605d0 con 0x7feb1815d0a0 2024-04-24T23:49:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.100+0000 7feb177fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb180ceca0 0x7feb1806ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.100+0000 7feb16ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7feb181455f0 0x7feb1806b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52322/0 (socket says 172.21.15.16:52322) 2024-04-24T23:49:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.100+0000 7feb16ffd640 1 -- 172.21.15.16:0/1724644156 learned_addr learned my addr 172.21.15.16:0/1724644156 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.100+0000 7feb0ffff640 1 --2- 172.21.15.16:0/1724644156 >> v2:172.21.15.156:3300/0 conn(0x7feb1815d0a0 0x7feb180ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.100+0000 7feb177fe640 1 -- 172.21.15.16:0/1724644156 >> v2:172.21.15.156:3300/0 conn(0x7feb1815d0a0 msgr2=0x7feb180ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.100+0000 7feb177fe640 1 --2- 172.21.15.16:0/1724644156 >> v2:172.21.15.156:3300/0 conn(0x7feb1815d0a0 0x7feb180ce760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.100+0000 7feb177fe640 1 -- 172.21.15.16:0/1724644156 >> v1:172.21.15.16:6789/0 conn(0x7feb181455f0 legacy=0x7feb1806b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.100+0000 7feb167fc640 1 -- 172.21.15.16:0/1724644156 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2922874098 0 0) 0x7feb1815f9c0 con 0x7feb181455f0 2024-04-24T23:49:43.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.100+0000 7feb177fe640 1 -- 172.21.15.16:0/1724644156 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb1815fca0 con 0x7feb180ceca0 2024-04-24T23:49:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.101+0000 7feb177fe640 1 --2- 172.21.15.16:0/1724644156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb180ceca0 0x7feb1806ccc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feb180cf340 tx=0x7feb1002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba809f3d3aad4cc6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.101+0000 7feb167fc640 1 -- 172.21.15.16:0/1724644156 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb10007740 con 0x7feb180ceca0 2024-04-24T23:49:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.101+0000 7feb167fc640 1 -- 172.21.15.16:0/1724644156 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb100078e0 con 0x7feb180ceca0 2024-04-24T23:49:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.101+0000 7feb167fc640 1 -- 172.21.15.16:0/1724644156 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb10007cb0 con 0x7feb180ceca0 2024-04-24T23:49:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.102+0000 7feb1d314640 1 -- 172.21.15.16:0/1724644156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb180ceca0 msgr2=0x7feb1806ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.102+0000 7feb1d314640 1 --2- 172.21.15.16:0/1724644156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb180ceca0 0x7feb1806ccc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7feb180cf340 tx=0x7feb1002a5f0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.102+0000 7feb1d314640 1 -- 172.21.15.16:0/1724644156 shutdown_connections 2024-04-24T23:49:43.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.102+0000 7feb1d314640 1 --2- 172.21.15.16:0/1724644156 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb180ceca0 0x7feb1806ccc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.102+0000 7feb1d314640 1 --2- 172.21.15.16:0/1724644156 >> v2:172.21.15.156:3300/0 conn(0x7feb1815d0a0 0x7feb180ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.102+0000 7feb1d314640 1 -- 172.21.15.16:0/1724644156 >> 172.21.15.16:0/1724644156 conn(0x7feb180cfc50 msgr2=0x7feb180d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.102+0000 7feb1d314640 1 -- 172.21.15.16:0/1724644156 shutdown_connections 2024-04-24T23:49:43.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.102+0000 7feb1d314640 1 -- 172.21.15.16:0/1724644156 wait complete. 2024-04-24T23:49:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.103+0000 7feb1d314640 1 Processor -- start 2024-04-24T23:49:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.103+0000 7feb1d314640 1 -- start start 2024-04-24T23:49:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb1d314640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb180ceca0 0x7feb181f52d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb1d314640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb181455f0 0x7feb181eea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb1d314640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb1815d0a0 0x7feb181eef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb1d314640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feb1806b7f0 con 0x7feb181455f0 2024-04-24T23:49:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb1d314640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7feb18160260 con 0x7feb180ceca0 2024-04-24T23:49:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb1d314640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7feb181459d0 con 0x7feb1815d0a0 2024-04-24T23:49:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb0ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb181455f0 0x7feb181eea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb0ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb181455f0 0x7feb181eea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59818/0 (socket says 172.21.15.16:59818) 2024-04-24T23:49:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb0ffff640 1 -- 172.21.15.16:0/112572996 learned_addr learned my addr 172.21.15.16:0/112572996 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb16ffd640 1 --2- 172.21.15.16:0/112572996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb180ceca0 0x7feb181f52d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb177fe640 1 --2- 172.21.15.16:0/112572996 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb1815d0a0 0x7feb181eef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb0ffff640 1 -- 172.21.15.16:0/112572996 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb1815d0a0 msgr2=0x7feb181eef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb0ffff640 1 --2- 172.21.15.16:0/112572996 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb1815d0a0 0x7feb181eef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb0ffff640 1 -- 172.21.15.16:0/112572996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb180ceca0 msgr2=0x7feb181f52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb0ffff640 1 --2- 172.21.15.16:0/112572996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb180ceca0 0x7feb181f52d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb0ffff640 1 -- 172.21.15.16:0/112572996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb18141ba0 con 0x7feb181455f0 2024-04-24T23:49:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb0ffff640 1 --2- 172.21.15.16:0/112572996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb181455f0 0x7feb181eea10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7feb0000d7d0 tx=0x7feb0000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb0f7fe640 1 -- 172.21.15.16:0/112572996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0000b660 con 0x7feb181455f0 2024-04-24T23:49:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb0f7fe640 1 -- 172.21.15.16:0/112572996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb00019070 con 0x7feb181455f0 2024-04-24T23:49:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb0f7fe640 1 -- 172.21.15.16:0/112572996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb0000ba20 con 0x7feb181455f0 2024-04-24T23:49:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.104+0000 7feb1d314640 1 -- 172.21.15.16:0/112572996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb18141e80 con 0x7feb181455f0 2024-04-24T23:49:43.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.105+0000 7feb1d314640 1 -- 172.21.15.16:0/112572996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb18142440 con 0x7feb181455f0 2024-04-24T23:49:43.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.105+0000 7feb1d314640 1 -- 172.21.15.16:0/112572996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb18068f90 con 0x7feb181455f0 2024-04-24T23:49:43.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.106+0000 7feb0f7fe640 1 -- 172.21.15.16:0/112572996 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7feb0000bbc0 con 0x7feb181455f0 2024-04-24T23:49:43.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.106+0000 7feb0f7fe640 1 --2- 172.21.15.16:0/112572996 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7feaf806be30 0x7feaf806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.106+0000 7feb0f7fe640 1 -- 172.21.15.16:0/112572996 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7feb0008b610 con 0x7feb181455f0 2024-04-24T23:49:43.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.106+0000 7feb16ffd640 1 --2- 172.21.15.16:0/112572996 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7feaf806be30 0x7feaf806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.107+0000 7feb16ffd640 1 --2- 172.21.15.16:0/112572996 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7feaf806be30 0x7feaf806e2f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feb08000f60 tx=0x7feb08007b50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.110+0000 7feb0f7fe640 1 -- 172.21.15.16:0/112572996 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7feb0001c360 con 0x7feb181455f0 2024-04-24T23:49:43.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.296+0000 7feb1d314640 1 -- 172.21.15.16:0/112572996 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7feb180b3150 con 0x7feb181455f0 2024-04-24T23:49:43.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.296+0000 7feb0f7fe640 1 -- 172.21.15.16:0/112572996 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7feb00059c50 con 0x7feb181455f0 2024-04-24T23:49:43.296 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.298+0000 7feb1d314640 1 -- 172.21.15.16:0/112572996 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7feaf806be30 msgr2=0x7feaf806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.298+0000 7feb1d314640 1 --2- 172.21.15.16:0/112572996 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7feaf806be30 0x7feaf806e2f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feb08000f60 tx=0x7feb08007b50 comp rx=0 tx=0).stop 2024-04-24T23:49:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.298+0000 7feb1d314640 1 -- 172.21.15.16:0/112572996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb181455f0 msgr2=0x7feb181eea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.298+0000 7feb1d314640 1 --2- 172.21.15.16:0/112572996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb181455f0 0x7feb181eea10 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7feb0000d7d0 tx=0x7feb0000dc90 comp rx=0 tx=0).stop 2024-04-24T23:49:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.298+0000 7feb1d314640 1 -- 172.21.15.16:0/112572996 shutdown_connections 2024-04-24T23:49:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.298+0000 7feb1d314640 1 --2- 172.21.15.16:0/112572996 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7feaf806be30 0x7feaf806e2f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.298+0000 7feb1d314640 1 --2- 172.21.15.16:0/112572996 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb1815d0a0 0x7feb181eef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.298+0000 7feb1d314640 1 --2- 172.21.15.16:0/112572996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb181455f0 0x7feb181eea10 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.298+0000 7feb1d314640 1 --2- 172.21.15.16:0/112572996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb180ceca0 0x7feb181f52d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.298+0000 7feb1d314640 1 -- 172.21.15.16:0/112572996 >> 172.21.15.16:0/112572996 conn(0x7feb180cfc50 msgr2=0x7feb1806b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.298+0000 7feb1d314640 1 -- 172.21.15.16:0/112572996 shutdown_connections 2024-04-24T23:49:43.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.298+0000 7feb1d314640 1 -- 172.21.15.16:0/112572996 wait complete. 2024-04-24T23:49:43.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:43.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.412+0000 7f3f704b3640 1 Processor -- start 2024-04-24T23:49:43.412 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.412+0000 7f3f704b3640 1 -- start start 2024-04-24T23:49:43.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.413+0000 7f3f704b3640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3f6806bc40 0x7f3f6806c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.413+0000 7f3f704b3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f680b7050 0x7f3f680b7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.413+0000 7f3f704b3640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f680d24d0 con 0x7f3f6815dfd0 2024-04-24T23:49:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.413+0000 7f3f704b3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f3f6806ba00 con 0x7f3f680b7050 2024-04-24T23:49:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.413+0000 7f3f704b3640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f3f680d27b0 con 0x7f3f6806bc40 2024-04-24T23:49:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.413+0000 7f3f6ea50640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f680b7050 0x7f3f680b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.414+0000 7f3f6ea50640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f680b7050 0x7f3f680b7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59826/0 (socket says 172.21.15.16:59826) 2024-04-24T23:49:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.414+0000 7f3f6ea50640 1 -- 172.21.15.16:0/196811786 learned_addr learned my addr 172.21.15.16:0/196811786 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.414+0000 7f3f6da4e640 1 --2- 172.21.15.16:0/196811786 >> v2:172.21.15.156:3300/0 conn(0x7f3f6806bc40 0x7f3f6806c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.414+0000 7f3f6ea50640 1 -- 172.21.15.16:0/196811786 >> v2:172.21.15.156:3300/0 conn(0x7f3f6806bc40 msgr2=0x7f3f6806c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.414+0000 7f3f6ea50640 1 --2- 172.21.15.16:0/196811786 >> v2:172.21.15.156:3300/0 conn(0x7f3f6806bc40 0x7f3f6806c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.414+0000 7f3f6ea50640 1 -- 172.21.15.16:0/196811786 >> v1:172.21.15.156:6789/0 conn(0x7f3f6815dfd0 legacy=0x7f3f6806b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.414+0000 7f3f6ea50640 1 -- 172.21.15.16:0/196811786 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f680d2a90 con 0x7f3f680b7050 2024-04-24T23:49:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.414+0000 7f3f6d24d640 1 -- 172.21.15.16:0/196811786 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4076478940 0 0) 0x7f3f680d24d0 con 0x7f3f6815dfd0 2024-04-24T23:49:43.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.415+0000 7f3f6ea50640 1 --2- 172.21.15.16:0/196811786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f680b7050 0x7f3f680b7460 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3f6806cf60 tx=0x7f3f6002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632e9eac86add158 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.415+0000 7f3f6d24d640 1 -- 172.21.15.16:0/196811786 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f60036070 con 0x7f3f680b7050 2024-04-24T23:49:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.415+0000 7f3f6d24d640 1 -- 172.21.15.16:0/196811786 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f60031600 con 0x7f3f680b7050 2024-04-24T23:49:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.415+0000 7f3f6d24d640 1 -- 172.21.15.16:0/196811786 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f600319d0 con 0x7f3f680b7050 2024-04-24T23:49:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.415+0000 7f3f704b3640 1 -- 172.21.15.16:0/196811786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f680b7050 msgr2=0x7f3f680b7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.415+0000 7f3f704b3640 1 --2- 172.21.15.16:0/196811786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f680b7050 0x7f3f680b7460 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3f6806cf60 tx=0x7f3f6002a6a0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.415+0000 7f3f704b3640 1 -- 172.21.15.16:0/196811786 shutdown_connections 2024-04-24T23:49:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.415+0000 7f3f704b3640 1 --2- 172.21.15.16:0/196811786 >> v2:172.21.15.156:3300/0 conn(0x7f3f6806bc40 0x7f3f6806c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.415+0000 7f3f704b3640 1 --2- 172.21.15.16:0/196811786 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f680b7050 0x7f3f680b7460 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.415+0000 7f3f704b3640 1 -- 172.21.15.16:0/196811786 >> 172.21.15.16:0/196811786 conn(0x7f3f680d3dc0 msgr2=0x7f3f680d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.415+0000 7f3f704b3640 1 -- 172.21.15.16:0/196811786 shutdown_connections 2024-04-24T23:49:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.415+0000 7f3f704b3640 1 -- 172.21.15.16:0/196811786 wait complete. 2024-04-24T23:49:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.416+0000 7f3f704b3640 1 Processor -- start 2024-04-24T23:49:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.416+0000 7f3f704b3640 1 -- start start 2024-04-24T23:49:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.417+0000 7f3f704b3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f6806bc40 0x7f3f681594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.417+0000 7f3f704b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f680b7050 0x7f3f6815a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.417+0000 7f3f704b3640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f6815dfd0 0x7f3f6815a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.417+0000 7f3f704b3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3f680cead0 con 0x7f3f680b7050 2024-04-24T23:49:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.417+0000 7f3f704b3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3f680d3050 con 0x7f3f6806bc40 2024-04-24T23:49:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.417+0000 7f3f704b3640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f3f680b79a0 con 0x7f3f6815dfd0 2024-04-24T23:49:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.417+0000 7f3f6da4e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f680b7050 0x7f3f6815a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.417+0000 7f3f6da4e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f680b7050 0x7f3f6815a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59828/0 (socket says 172.21.15.16:59828) 2024-04-24T23:49:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.417+0000 7f3f6da4e640 1 -- 172.21.15.16:0/369070904 learned_addr learned my addr 172.21.15.16:0/369070904 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.417+0000 7f3f6e24f640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f6806bc40 0x7f3f681594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.417+0000 7f3f6ea50640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f6815dfd0 0x7f3f6815a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.418+0000 7f3f6da4e640 1 -- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f6815dfd0 msgr2=0x7f3f6815a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.418+0000 7f3f6da4e640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f6815dfd0 0x7f3f6815a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.418+0000 7f3f6da4e640 1 -- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f6806bc40 msgr2=0x7f3f681594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.418+0000 7f3f6da4e640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f6806bc40 0x7f3f681594a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.418+0000 7f3f6da4e640 1 -- 172.21.15.16:0/369070904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f680caf30 con 0x7f3f680b7050 2024-04-24T23:49:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.418+0000 7f3f6ea50640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f6815dfd0 0x7f3f6815a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.418+0000 7f3f6e24f640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f6806bc40 0x7f3f681594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.418+0000 7f3f6da4e640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f680b7050 0x7f3f6815a1a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800e9b0 tx=0x7f3f5800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.418+0000 7f3f5effd640 1 -- 172.21.15.16:0/369070904 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f58017070 con 0x7f3f680b7050 2024-04-24T23:49:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.418+0000 7f3f5effd640 1 -- 172.21.15.16:0/369070904 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f5800c9a0 con 0x7f3f680b7050 2024-04-24T23:49:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.418+0000 7f3f704b3640 1 -- 172.21.15.16:0/369070904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f680cb1d0 con 0x7f3f680b7050 2024-04-24T23:49:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.418+0000 7f3f5effd640 1 -- 172.21.15.16:0/369070904 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f5800cd90 con 0x7f3f680b7050 2024-04-24T23:49:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.418+0000 7f3f704b3640 1 -- 172.21.15.16:0/369070904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f680cb790 con 0x7f3f680b7050 2024-04-24T23:49:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.419+0000 7f3f704b3640 1 -- 172.21.15.16:0/369070904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f68159f80 con 0x7f3f680b7050 2024-04-24T23:49:43.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.419+0000 7f3f5effd640 1 -- 172.21.15.16:0/369070904 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3f5801a070 con 0x7f3f680b7050 2024-04-24T23:49:43.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.420+0000 7f3f5effd640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3f4c06be30 0x7f3f4c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.420+0000 7f3f5effd640 1 -- 172.21.15.16:0/369070904 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f3f5808b210 con 0x7f3f680b7050 2024-04-24T23:49:43.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.420+0000 7f3f6e24f640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3f4c06be30 0x7f3f4c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.421+0000 7f3f6e24f640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3f4c06be30 0x7f3f4c06e2f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3f64000f60 tx=0x7f3f64009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:43.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.424+0000 7f3f5effd640 1 -- 172.21.15.16:0/369070904 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3f58059850 con 0x7f3f680b7050 2024-04-24T23:49:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.612+0000 7f3f704b3640 1 -- 172.21.15.16:0/369070904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3f68068f90 con 0x7f3f680b7050 2024-04-24T23:49:43.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.612+0000 7f3f5effd640 1 -- 172.21.15.16:0/369070904 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f3f58059150 con 0x7f3f680b7050 2024-04-24T23:49:43.612 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.614+0000 7f3f704b3640 1 -- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3f4c06be30 msgr2=0x7f3f4c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.614+0000 7f3f704b3640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3f4c06be30 0x7f3f4c06e2f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3f64000f60 tx=0x7f3f64009040 comp rx=0 tx=0).stop 2024-04-24T23:49:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.614+0000 7f3f704b3640 1 -- 172.21.15.16:0/369070904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f680b7050 msgr2=0x7f3f6815a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.614+0000 7f3f704b3640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f680b7050 0x7f3f6815a1a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800e9b0 tx=0x7f3f5800ee70 comp rx=0 tx=0).stop 2024-04-24T23:49:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.614+0000 7f3f704b3640 1 -- 172.21.15.16:0/369070904 shutdown_connections 2024-04-24T23:49:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.614+0000 7f3f704b3640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3f4c06be30 0x7f3f4c06e2f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.614+0000 7f3f704b3640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3f6815dfd0 0x7f3f6815a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.614+0000 7f3f704b3640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f680b7050 0x7f3f6815a1a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.614+0000 7f3f704b3640 1 --2- 172.21.15.16:0/369070904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3f6806bc40 0x7f3f681594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.614+0000 7f3f704b3640 1 -- 172.21.15.16:0/369070904 >> 172.21.15.16:0/369070904 conn(0x7f3f680d3dc0 msgr2=0x7f3f680cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.614+0000 7f3f704b3640 1 -- 172.21.15.16:0/369070904 shutdown_connections 2024-04-24T23:49:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.614+0000 7f3f704b3640 1 -- 172.21.15.16:0/369070904 wait complete. 2024-04-24T23:49:43.625 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.733+0000 7f9e2df63640 1 Processor -- start 2024-04-24T23:49:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.733+0000 7f9e2df63640 1 -- start start 2024-04-24T23:49:43.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.734+0000 7f9e2df63640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9e280d12d0 0x7f9e280d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.734+0000 7f9e2df63640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e2815e5c0 0x7f9e280d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.734+0000 7f9e2df63640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e2815ec30 con 0x7f9e280cd790 2024-04-24T23:49:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.734+0000 7f9e2df63640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f9e2815f4d0 con 0x7f9e280d12d0 2024-04-24T23:49:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.734+0000 7f9e2df63640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f9e2815ef10 con 0x7f9e2815e5c0 2024-04-24T23:49:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.734+0000 7f9e27fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e2815e5c0 0x7f9e280d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.734+0000 7f9e277fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9e280d12d0 0x7f9e280d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.735+0000 7f9e26ffd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f9e280cd790 0x7f9e280c8580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:53016/0 (socket says 172.21.15.16:53016) 2024-04-24T23:49:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.735+0000 7f9e27fff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e2815e5c0 0x7f9e280d2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:53540/0 (socket says 172.21.15.16:53540) 2024-04-24T23:49:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.735+0000 7f9e26ffd640 1 -- 172.21.15.16:0/1814186579 learned_addr learned my addr 172.21.15.16:0/1814186579 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.735+0000 7f9e277fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9e280d12d0 0x7f9e280d16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:39880/0 (socket says 172.21.15.16:39880) 2024-04-24T23:49:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.735+0000 7f9e27fff640 1 -- 172.21.15.16:0/1814186579 >> v2:172.21.15.156:3300/0 conn(0x7f9e280d12d0 msgr2=0x7f9e280d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.735+0000 7f9e27fff640 1 --2- 172.21.15.16:0/1814186579 >> v2:172.21.15.156:3300/0 conn(0x7f9e280d12d0 0x7f9e280d16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.735+0000 7f9e27fff640 1 -- 172.21.15.16:0/1814186579 >> v1:172.21.15.156:6789/0 conn(0x7f9e280cd790 legacy=0x7f9e280c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.735+0000 7f9e27fff640 1 -- 172.21.15.16:0/1814186579 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e2815f1f0 con 0x7f9e2815e5c0 2024-04-24T23:49:43.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.735+0000 7f9e277fe640 1 --2- 172.21.15.16:0/1814186579 >> v2:172.21.15.156:3300/0 conn(0x7f9e280d12d0 0x7f9e280d16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.736+0000 7f9e27fff640 1 --2- 172.21.15.16:0/1814186579 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e2815e5c0 0x7f9e280d2130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e2815f650 tx=0x7f9e2002a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a15cdebdaad60dfc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.736+0000 7f9e267fc640 1 -- 172.21.15.16:0/1814186579 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20007740 con 0x7f9e2815e5c0 2024-04-24T23:49:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.736+0000 7f9e267fc640 1 -- 172.21.15.16:0/1814186579 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e200078e0 con 0x7f9e2815e5c0 2024-04-24T23:49:43.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.736+0000 7f9e267fc640 1 -- 172.21.15.16:0/1814186579 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e20007cb0 con 0x7f9e2815e5c0 2024-04-24T23:49:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.736+0000 7f9e2df63640 1 -- 172.21.15.16:0/1814186579 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e2815e5c0 msgr2=0x7f9e280d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.736+0000 7f9e2df63640 1 --2- 172.21.15.16:0/1814186579 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e2815e5c0 0x7f9e280d2130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9e2815f650 tx=0x7f9e2002a530 comp rx=0 tx=0).stop 2024-04-24T23:49:43.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.736+0000 7f9e2df63640 1 -- 172.21.15.16:0/1814186579 shutdown_connections 2024-04-24T23:49:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.736+0000 7f9e2df63640 1 --2- 172.21.15.16:0/1814186579 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e2815e5c0 0x7f9e280d2130 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.736+0000 7f9e2df63640 1 --2- 172.21.15.16:0/1814186579 >> v2:172.21.15.156:3300/0 conn(0x7f9e280d12d0 0x7f9e280d16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.736+0000 7f9e2df63640 1 -- 172.21.15.16:0/1814186579 >> 172.21.15.16:0/1814186579 conn(0x7f9e28072e70 msgr2=0x7f9e28073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.736+0000 7f9e2df63640 1 -- 172.21.15.16:0/1814186579 shutdown_connections 2024-04-24T23:49:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.736+0000 7f9e2df63640 1 -- 172.21.15.16:0/1814186579 wait complete. 2024-04-24T23:49:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.738+0000 7f9e2df63640 1 Processor -- start 2024-04-24T23:49:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.738+0000 7f9e2df63640 1 -- start start 2024-04-24T23:49:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.738+0000 7f9e2df63640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e280cd790 0x7f9e280afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.738+0000 7f9e2df63640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e280d12d0 0x7f9e280ae540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.738+0000 7f9e2df63640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e2815e5c0 0x7f9e280af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.738+0000 7f9e2df63640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9e280d27f0 con 0x7f9e280d12d0 2024-04-24T23:49:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.738+0000 7f9e2df63640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9e280d2670 con 0x7f9e280cd790 2024-04-24T23:49:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.738+0000 7f9e2df63640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f9e280d2970 con 0x7f9e2815e5c0 2024-04-24T23:49:43.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e26ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e280d12d0 0x7f9e280ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e26ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e280d12d0 0x7f9e280ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59842/0 (socket says 172.21.15.16:59842) 2024-04-24T23:49:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e26ffd640 1 -- 172.21.15.16:0/2653305297 learned_addr learned my addr 172.21.15.16:0/2653305297 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e27fff640 1 --2- 172.21.15.16:0/2653305297 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e2815e5c0 0x7f9e280af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e277fe640 1 --2- 172.21.15.16:0/2653305297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e280cd790 0x7f9e280afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e26ffd640 1 -- 172.21.15.16:0/2653305297 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e2815e5c0 msgr2=0x7f9e280af070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.741 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e26ffd640 1 --2- 172.21.15.16:0/2653305297 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e2815e5c0 0x7f9e280af070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e26ffd640 1 -- 172.21.15.16:0/2653305297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e280cd790 msgr2=0x7f9e280afe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e26ffd640 1 --2- 172.21.15.16:0/2653305297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e280cd790 0x7f9e280afe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e26ffd640 1 -- 172.21.15.16:0/2653305297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e280b0370 con 0x7f9e280d12d0 2024-04-24T23:49:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e26ffd640 1 --2- 172.21.15.16:0/2653305297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e280d12d0 0x7f9e280ae540 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00c480 tx=0x7f9e1c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e0ffff640 1 -- 172.21.15.16:0/2653305297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c017070 con 0x7f9e280d12d0 2024-04-24T23:49:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e0ffff640 1 -- 172.21.15.16:0/2653305297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9e1c005ce0 con 0x7f9e280d12d0 2024-04-24T23:49:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e2df63640 1 -- 172.21.15.16:0/2653305297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e280b0610 con 0x7f9e280d12d0 2024-04-24T23:49:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e0ffff640 1 -- 172.21.15.16:0/2653305297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e1c0072f0 con 0x7f9e280d12d0 2024-04-24T23:49:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.739+0000 7f9e2df63640 1 -- 172.21.15.16:0/2653305297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e280add40 con 0x7f9e280d12d0 2024-04-24T23:49:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.740+0000 7f9e2df63640 1 -- 172.21.15.16:0/2653305297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e28068f90 con 0x7f9e280d12d0 2024-04-24T23:49:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.741+0000 7f9e0ffff640 1 -- 172.21.15.16:0/2653305297 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9e1c01a070 con 0x7f9e280d12d0 2024-04-24T23:49:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.741+0000 7f9e0ffff640 1 --2- 172.21.15.16:0/2653305297 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9e1406be30 0x7f9e1406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.741+0000 7f9e0ffff640 1 -- 172.21.15.16:0/2653305297 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f9e1c08a460 con 0x7f9e280d12d0 2024-04-24T23:49:43.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.741+0000 7f9e277fe640 1 --2- 172.21.15.16:0/2653305297 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9e1406be30 0x7f9e1406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:43.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.742+0000 7f9e277fe640 1 --2- 172.21.15.16:0/2653305297 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9e1406be30 0x7f9e1406e2f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009560 tx=0x7f9e18009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:43.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.746+0000 7f9e0ffff640 1 -- 172.21.15.16:0/2653305297 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9e1c058450 con 0x7f9e280d12d0 2024-04-24T23:49:43.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.927+0000 7f9e2df63640 1 -- 172.21.15.16:0/2653305297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9e280afc10 con 0x7f9e280d12d0 2024-04-24T23:49:43.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.927+0000 7f9e0ffff640 1 -- 172.21.15.16:0/2653305297 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f9e1c05c750 con 0x7f9e280d12d0 2024-04-24T23:49:43.927 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:43.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.929+0000 7f9e2df63640 1 -- 172.21.15.16:0/2653305297 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9e1406be30 msgr2=0x7f9e1406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.929+0000 7f9e2df63640 1 --2- 172.21.15.16:0/2653305297 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9e1406be30 0x7f9e1406e2f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9e18009560 tx=0x7f9e18009190 comp rx=0 tx=0).stop 2024-04-24T23:49:43.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.929+0000 7f9e2df63640 1 -- 172.21.15.16:0/2653305297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e280d12d0 msgr2=0x7f9e280ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:43.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.929+0000 7f9e2df63640 1 --2- 172.21.15.16:0/2653305297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e280d12d0 0x7f9e280ae540 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f9e1c00c480 tx=0x7f9e1c00c940 comp rx=0 tx=0).stop 2024-04-24T23:49:43.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.929+0000 7f9e2df63640 1 -- 172.21.15.16:0/2653305297 shutdown_connections 2024-04-24T23:49:43.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.929+0000 7f9e2df63640 1 --2- 172.21.15.16:0/2653305297 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9e1406be30 0x7f9e1406e2f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.929+0000 7f9e2df63640 1 --2- 172.21.15.16:0/2653305297 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9e2815e5c0 0x7f9e280af070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.929+0000 7f9e2df63640 1 --2- 172.21.15.16:0/2653305297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e280d12d0 0x7f9e280ae540 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.929+0000 7f9e2df63640 1 --2- 172.21.15.16:0/2653305297 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9e280cd790 0x7f9e280afe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:43.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.929+0000 7f9e2df63640 1 -- 172.21.15.16:0/2653305297 >> 172.21.15.16:0/2653305297 conn(0x7f9e28072e70 msgr2=0x7f9e280cfd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:43.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.929+0000 7f9e2df63640 1 -- 172.21.15.16:0/2653305297 shutdown_connections 2024-04-24T23:49:43.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:43.929+0000 7f9e2df63640 1 -- 172.21.15.16:0/2653305297 wait complete. 2024-04-24T23:49:43.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.044+0000 7f1779212640 1 Processor -- start 2024-04-24T23:49:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.044+0000 7f1779212640 1 -- start start 2024-04-24T23:49:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.045+0000 7f1779212640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f177415e020 0x7f17740cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.045+0000 7f1779212640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f17740d1fa0 0x7f17740d2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.045+0000 7f1779212640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17740b3390 con 0x7f17740b2bb0 2024-04-24T23:49:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.045+0000 7f1779212640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f177406bfc0 con 0x7f17740d1fa0 2024-04-24T23:49:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.045+0000 7f1779212640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f177406c2a0 con 0x7f177415e020 2024-04-24T23:49:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.045+0000 7f177359e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f17740d1fa0 0x7f17740d2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.045+0000 7f177259c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f17740b2bb0 0x7f17740cf7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52324/0 (socket says 172.21.15.16:52324) 2024-04-24T23:49:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.045+0000 7f177259c640 1 -- 172.21.15.16:0/72708175 learned_addr learned my addr 172.21.15.16:0/72708175 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.045+0000 7f1772d9d640 1 --2- 172.21.15.16:0/72708175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f177415e020 0x7f17740cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.045+0000 7f177359e640 1 -- 172.21.15.16:0/72708175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f177415e020 msgr2=0x7f17740cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.046+0000 7f177359e640 1 --2- 172.21.15.16:0/72708175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f177415e020 0x7f17740cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.046+0000 7f177359e640 1 -- 172.21.15.16:0/72708175 >> v1:172.21.15.16:6789/0 conn(0x7f17740b2bb0 legacy=0x7f17740cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.046+0000 7f177359e640 1 -- 172.21.15.16:0/72708175 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177406c580 con 0x7f17740d1fa0 2024-04-24T23:49:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.046+0000 7f1771d9b640 1 -- 172.21.15.16:0/72708175 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3203110285 0 0) 0x7f17740b3390 con 0x7f17740b2bb0 2024-04-24T23:49:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.046+0000 7f1772d9d640 1 --2- 172.21.15.16:0/72708175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f177415e020 0x7f17740cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.046+0000 7f177359e640 1 --2- 172.21.15.16:0/72708175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17740d1fa0 0x7f17740d2410 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f176c0097a0 tx=0x7f176c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72c2413a138946b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.046+0000 7f1771d9b640 1 -- 172.21.15.16:0/72708175 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c037070 con 0x7f17740d1fa0 2024-04-24T23:49:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.046+0000 7f1771d9b640 1 -- 172.21.15.16:0/72708175 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f176c02ae50 con 0x7f17740d1fa0 2024-04-24T23:49:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.046+0000 7f1771d9b640 1 -- 172.21.15.16:0/72708175 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176c032660 con 0x7f17740d1fa0 2024-04-24T23:49:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.047+0000 7f1779212640 1 -- 172.21.15.16:0/72708175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17740d1fa0 msgr2=0x7f17740d2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.047+0000 7f1779212640 1 --2- 172.21.15.16:0/72708175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17740d1fa0 0x7f17740d2410 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f176c0097a0 tx=0x7f176c02a100 comp rx=0 tx=0).stop 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.047+0000 7f1779212640 1 -- 172.21.15.16:0/72708175 shutdown_connections 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.047+0000 7f1779212640 1 --2- 172.21.15.16:0/72708175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17740d1fa0 0x7f17740d2410 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.047+0000 7f1779212640 1 --2- 172.21.15.16:0/72708175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f177415e020 0x7f17740cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.047+0000 7f1779212640 1 -- 172.21.15.16:0/72708175 >> 172.21.15.16:0/72708175 conn(0x7f17740d3e10 msgr2=0x7f17740d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.047+0000 7f1779212640 1 -- 172.21.15.16:0/72708175 shutdown_connections 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.047+0000 7f1779212640 1 -- 172.21.15.16:0/72708175 wait complete. 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.048+0000 7f1779212640 1 Processor -- start 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.048+0000 7f1779212640 1 -- start start 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.049+0000 7f1779212640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17740b2bb0 0x7f1774157350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.049+0000 7f1779212640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17740d1fa0 0x7f1774160960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.049+0000 7f1779212640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f177415e020 0x7f17740b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.049+0000 7f1779212640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f177415e400 con 0x7f17740b2bb0 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.049+0000 7f1779212640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f17740d1a80 con 0x7f177415e020 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.049+0000 7f1779212640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f17740cfec0 con 0x7f17740d1fa0 2024-04-24T23:49:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.049+0000 7f1772d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17740b2bb0 0x7f1774157350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.049+0000 7f1772d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17740b2bb0 0x7f1774157350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59858/0 (socket says 172.21.15.16:59858) 2024-04-24T23:49:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.049+0000 7f1772d9d640 1 -- 172.21.15.16:0/898965325 learned_addr learned my addr 172.21.15.16:0/898965325 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.049+0000 7f177259c640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17740d1fa0 0x7f1774160960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.049+0000 7f177359e640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f177415e020 0x7f17740b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.050+0000 7f1772d9d640 1 -- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17740d1fa0 msgr2=0x7f1774160960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.050+0000 7f1772d9d640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17740d1fa0 0x7f1774160960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.050+0000 7f1772d9d640 1 -- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f177415e020 msgr2=0x7f17740b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.050+0000 7f1772d9d640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f177415e020 0x7f17740b4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.050+0000 7f1772d9d640 1 -- 172.21.15.16:0/898965325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17741f81d0 con 0x7f17740b2bb0 2024-04-24T23:49:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.050+0000 7f177359e640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f177415e020 0x7f17740b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.050+0000 7f1772d9d640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17740b2bb0 0x7f1774157350 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f176400cf60 tx=0x7f176400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.050+0000 7f176b7fe640 1 -- 172.21.15.16:0/898965325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176401a070 con 0x7f17740b2bb0 2024-04-24T23:49:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.050+0000 7f176b7fe640 1 -- 172.21.15.16:0/898965325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f176400d840 con 0x7f17740b2bb0 2024-04-24T23:49:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.050+0000 7f1779212640 1 -- 172.21.15.16:0/898965325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17741f7410 con 0x7f17740b2bb0 2024-04-24T23:49:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.050+0000 7f176b7fe640 1 -- 172.21.15.16:0/898965325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176400dc10 con 0x7f17740b2bb0 2024-04-24T23:49:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.050+0000 7f1779212640 1 -- 172.21.15.16:0/898965325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17741f79d0 con 0x7f17740b2bb0 2024-04-24T23:49:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.051+0000 7f1779212640 1 -- 172.21.15.16:0/898965325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f177406dfe0 con 0x7f17740b2bb0 2024-04-24T23:49:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.052+0000 7f176b7fe640 1 -- 172.21.15.16:0/898965325 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f176401d070 con 0x7f17740b2bb0 2024-04-24T23:49:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.052+0000 7f176b7fe640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f175406be60 0x7f175406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.052+0000 7f176b7fe640 1 -- 172.21.15.16:0/898965325 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f176408b930 con 0x7f17740b2bb0 2024-04-24T23:49:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.052+0000 7f177259c640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f175406be60 0x7f175406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.053+0000 7f177259c640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f175406be60 0x7f175406e320 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f17740cc0c0 tx=0x7f175c007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:44.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.056+0000 7f176b7fe640 1 -- 172.21.15.16:0/898965325 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1764059920 con 0x7f17740b2bb0 2024-04-24T23:49:44.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.239+0000 7f1779212640 1 -- 172.21.15.16:0/898965325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f177404f5e0 con 0x7f17740b2bb0 2024-04-24T23:49:44.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.240+0000 7f176b7fe640 1 -- 172.21.15.16:0/898965325 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7f176405dc20 con 0x7f17740b2bb0 2024-04-24T23:49:44.240 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:44.240 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:44.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.242+0000 7f1779212640 1 -- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f175406be60 msgr2=0x7f175406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.242+0000 7f1779212640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f175406be60 0x7f175406e320 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f17740cc0c0 tx=0x7f175c007c00 comp rx=0 tx=0).stop 2024-04-24T23:49:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.242+0000 7f1779212640 1 -- 172.21.15.16:0/898965325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17740b2bb0 msgr2=0x7f1774157350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.242+0000 7f1779212640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17740b2bb0 0x7f1774157350 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f176400cf60 tx=0x7f176400cf90 comp rx=0 tx=0).stop 2024-04-24T23:49:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.242+0000 7f1779212640 1 -- 172.21.15.16:0/898965325 shutdown_connections 2024-04-24T23:49:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.242+0000 7f1779212640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f175406be60 0x7f175406e320 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.242+0000 7f1779212640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f177415e020 0x7f17740b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.242+0000 7f1779212640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f17740d1fa0 0x7f1774160960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.242+0000 7f1779212640 1 --2- 172.21.15.16:0/898965325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f17740b2bb0 0x7f1774157350 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.242+0000 7f1779212640 1 -- 172.21.15.16:0/898965325 >> 172.21.15.16:0/898965325 conn(0x7f17740d3e10 msgr2=0x7f17740b6f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.242+0000 7f1779212640 1 -- 172.21.15.16:0/898965325 shutdown_connections 2024-04-24T23:49:44.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.242+0000 7f1779212640 1 -- 172.21.15.16:0/898965325 wait complete. 2024-04-24T23:49:44.253 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:44.254 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:44.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.358+0000 7f7ad309e640 1 Processor -- start 2024-04-24T23:49:44.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.358+0000 7f7ad309e640 1 -- start start 2024-04-24T23:49:44.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.359+0000 7f7ad309e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7acc15e070 0x7f7acc06c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.359+0000 7f7ad309e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7acc15e450 con 0x7f7acc0b6f90 2024-04-24T23:49:44.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.359+0000 7f7ad309e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7acc0ce790 con 0x7f7acc0d3050 2024-04-24T23:49:44.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.359+0000 7f7ad309e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f7acc0cf3a0 con 0x7f7acc15e070 2024-04-24T23:49:44.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.359+0000 7f7ad0e3a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7acc15e070 0x7f7acc06c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.359+0000 7f7acbfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7acc0b6f90 0x7f7acc0d2940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52338/0 (socket says 172.21.15.16:52338) 2024-04-24T23:49:44.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.359+0000 7f7acbfff640 1 -- 172.21.15.16:0/3666765621 learned_addr learned my addr 172.21.15.16:0/3666765621 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:44.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.359+0000 7f7ad0e3a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7acc15e070 0x7f7acc06c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59874/0 (socket says 172.21.15.16:59874) 2024-04-24T23:49:44.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.360+0000 7f7ad0e3a640 1 -- 172.21.15.16:0/3666765621 >> v1:172.21.15.156:6789/0 conn(0x7f7acc0d3050 legacy=0x7f7acc0cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.360+0000 7f7ad0e3a640 1 -- 172.21.15.16:0/3666765621 >> v1:172.21.15.16:6789/0 conn(0x7f7acc0b6f90 legacy=0x7f7acc0d2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.360+0000 7f7ad0e3a640 1 -- 172.21.15.16:0/3666765621 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7acc0cea70 con 0x7f7acc15e070 2024-04-24T23:49:44.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.360+0000 7f7acb7fe640 1 -- 172.21.15.16:0/3666765621 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 293692580 0 0) 0x7f7acc15e450 con 0x7f7acc0b6f90 2024-04-24T23:49:44.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.360+0000 7f7ad0e3a640 1 --2- 172.21.15.16:0/3666765621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc15e070 0x7f7acc06c330 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4009d90 tx=0x7f7ac402a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c6cae01b4a9f15 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:44.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.360+0000 7f7acb7fe640 1 -- 172.21.15.16:0/3666765621 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4007740 con 0x7f7acc15e070 2024-04-24T23:49:44.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.360+0000 7f7acb7fe640 1 -- 172.21.15.16:0/3666765621 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ac40078e0 con 0x7f7acc15e070 2024-04-24T23:49:44.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.360+0000 7f7acb7fe640 1 -- 172.21.15.16:0/3666765621 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4007cb0 con 0x7f7acc15e070 2024-04-24T23:49:44.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.361+0000 7f7ad309e640 1 -- 172.21.15.16:0/3666765621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc15e070 msgr2=0x7f7acc06c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.361+0000 7f7ad309e640 1 --2- 172.21.15.16:0/3666765621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc15e070 0x7f7acc06c330 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4009d90 tx=0x7f7ac402a660 comp rx=0 tx=0).stop 2024-04-24T23:49:44.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.361+0000 7f7ad309e640 1 -- 172.21.15.16:0/3666765621 shutdown_connections 2024-04-24T23:49:44.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.361+0000 7f7ad309e640 1 --2- 172.21.15.16:0/3666765621 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc15e070 0x7f7acc06c330 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.361+0000 7f7ad309e640 1 -- 172.21.15.16:0/3666765621 >> 172.21.15.16:0/3666765621 conn(0x7f7acc0b5850 msgr2=0x7f7acc0b5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:44.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.361+0000 7f7ad309e640 1 -- 172.21.15.16:0/3666765621 shutdown_connections 2024-04-24T23:49:44.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.361+0000 7f7ad309e640 1 -- 172.21.15.16:0/3666765621 wait complete. 2024-04-24T23:49:44.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.362+0000 7f7ad309e640 1 Processor -- start 2024-04-24T23:49:44.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.362+0000 7f7ad309e640 1 -- start start 2024-04-24T23:49:44.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7ad309e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7acc0b6f90 0x7f7acc14c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7ad309e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc0d3050 0x7f7acc0cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7ad309e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7acc15e070 0x7f7acc15e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7ad309e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc0ce3b0 con 0x7f7acc0d3050 2024-04-24T23:49:44.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7ad309e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7acc0cf030 con 0x7f7acc0b6f90 2024-04-24T23:49:44.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7ad309e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7acc0ce530 con 0x7f7acc15e070 2024-04-24T23:49:44.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7acbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc0d3050 0x7f7acc0cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7acbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc0d3050 0x7f7acc0cbe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59884/0 (socket says 172.21.15.16:59884) 2024-04-24T23:49:44.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7acbfff640 1 -- 172.21.15.16:0/863529448 learned_addr learned my addr 172.21.15.16:0/863529448 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:44.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7ad0e3a640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7acc0b6f90 0x7f7acc14c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7ad163b640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7acc15e070 0x7f7acc15e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7acbfff640 1 -- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7acc15e070 msgr2=0x7f7acc15e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7acbfff640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7acc15e070 0x7f7acc15e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7acbfff640 1 -- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7acc0b6f90 msgr2=0x7f7acc14c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.363+0000 7f7acbfff640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7acc0b6f90 0x7f7acc14c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.364+0000 7f7acbfff640 1 -- 172.21.15.16:0/863529448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7acc0cc580 con 0x7f7acc0d3050 2024-04-24T23:49:44.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.364+0000 7f7ad0e3a640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7acc0b6f90 0x7f7acc14c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:44.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.364+0000 7f7ad163b640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7acc15e070 0x7f7acc15e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:44.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.364+0000 7f7acbfff640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc0d3050 0x7f7acc0cbe80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000c3f0 tx=0x7f7ac000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:44.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.364+0000 7f7ac97fa640 1 -- 172.21.15.16:0/863529448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac00074e0 con 0x7f7acc0d3050 2024-04-24T23:49:44.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.364+0000 7f7ac97fa640 1 -- 172.21.15.16:0/863529448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ac00047e0 con 0x7f7acc0d3050 2024-04-24T23:49:44.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.364+0000 7f7ad309e640 1 -- 172.21.15.16:0/863529448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7acc0cc860 con 0x7f7acc0d3050 2024-04-24T23:49:44.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.364+0000 7f7ac97fa640 1 -- 172.21.15.16:0/863529448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ac0004c10 con 0x7f7acc0d3050 2024-04-24T23:49:44.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.364+0000 7f7ad309e640 1 -- 172.21.15.16:0/863529448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7acc0cce20 con 0x7f7acc0d3050 2024-04-24T23:49:44.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.365+0000 7f7ad309e640 1 -- 172.21.15.16:0/863529448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7acc068f90 con 0x7f7acc0d3050 2024-04-24T23:49:44.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.365+0000 7f7ac97fa640 1 -- 172.21.15.16:0/863529448 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7ac0011070 con 0x7f7acc0d3050 2024-04-24T23:49:44.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.366+0000 7f7ac97fa640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7ab806be60 0x7f7ab806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.366+0000 7f7ac97fa640 1 -- 172.21.15.16:0/863529448 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7ac008aeb0 con 0x7f7acc0d3050 2024-04-24T23:49:44.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.366+0000 7f7ad0e3a640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7ab806be60 0x7f7ab806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.366+0000 7f7ad0e3a640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7ab806be60 0x7f7ab806e320 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4034000 tx=0x7f7ac402f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:44.369 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.370+0000 7f7ac97fa640 1 -- 172.21.15.16:0/863529448 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7ac001a2b0 con 0x7f7acc0d3050 2024-04-24T23:49:44.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.551+0000 7f7ad309e640 1 -- 172.21.15.16:0/863529448 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7acc0b2f00 con 0x7f7acc0d3050 2024-04-24T23:49:44.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.552+0000 7f7ac97fa640 1 -- 172.21.15.16:0/863529448 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f7ac0059460 con 0x7f7acc0d3050 2024-04-24T23:49:44.552 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:44.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.554+0000 7f7ad309e640 1 -- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7ab806be60 msgr2=0x7f7ab806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.554+0000 7f7ad309e640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7ab806be60 0x7f7ab806e320 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7ac4034000 tx=0x7f7ac402f000 comp rx=0 tx=0).stop 2024-04-24T23:49:44.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.554+0000 7f7ad309e640 1 -- 172.21.15.16:0/863529448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc0d3050 msgr2=0x7f7acc0cbe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.554+0000 7f7ad309e640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc0d3050 0x7f7acc0cbe80 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7ac000c3f0 tx=0x7f7ac000c8b0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.554+0000 7f7ad309e640 1 -- 172.21.15.16:0/863529448 shutdown_connections 2024-04-24T23:49:44.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.554+0000 7f7ad309e640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7ab806be60 0x7f7ab806e320 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.554+0000 7f7ad309e640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7acc15e070 0x7f7acc15e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.554+0000 7f7ad309e640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7acc0d3050 0x7f7acc0cbe80 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.554+0000 7f7ad309e640 1 --2- 172.21.15.16:0/863529448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7acc0b6f90 0x7f7acc14c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.554+0000 7f7ad309e640 1 -- 172.21.15.16:0/863529448 >> 172.21.15.16:0/863529448 conn(0x7f7acc0b5850 msgr2=0x7f7acc06c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:44.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.554+0000 7f7ad309e640 1 -- 172.21.15.16:0/863529448 shutdown_connections 2024-04-24T23:49:44.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.554+0000 7f7ad309e640 1 -- 172.21.15.16:0/863529448 wait complete. 2024-04-24T23:49:44.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:44.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.676+0000 7f7e120a5640 1 Processor -- start 2024-04-24T23:49:44.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.676+0000 7f7e120a5640 1 -- start start 2024-04-24T23:49:44.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e120a5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7e0c072a30 0x7f7e0c165300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e120a5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e0c15acd0 con 0x7f7e0c154ba0 2024-04-24T23:49:44.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e120a5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e0c15afb0 con 0x7f7e0c0b3750 2024-04-24T23:49:44.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e120a5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f7e0c0b5740 con 0x7f7e0c072a30 2024-04-24T23:49:44.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e0b7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7e0c154ba0 0x7f7e0c0b5030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52348/0 (socket says 172.21.15.16:52348) 2024-04-24T23:49:44.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e0b7fe640 1 -- 172.21.15.16:0/3713713533 learned_addr learned my addr 172.21.15.16:0/3713713533 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:44.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e0bfff640 1 --2- 172.21.15.16:0/3713713533 >> v2:172.21.15.156:3300/0 conn(0x7f7e0c072a30 0x7f7e0c165300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e0a7fc640 1 -- 172.21.15.16:0/3713713533 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1780049810 0 0) 0x7f7e0c15acd0 con 0x7f7e0c154ba0 2024-04-24T23:49:44.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e0a7fc640 1 -- 172.21.15.16:0/3713713533 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7df4003520 con 0x7f7e0c154ba0 2024-04-24T23:49:44.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e0a7fc640 1 -- 172.21.15.16:0/3713713533 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2238702790 0 0) 0x7f7df4003520 con 0x7f7e0c154ba0 2024-04-24T23:49:44.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e0a7fc640 1 -- 172.21.15.16:0/3713713533 >> v2:172.21.15.156:3300/0 conn(0x7f7e0c072a30 msgr2=0x7f7e0c165300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:44.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e0a7fc640 1 --2- 172.21.15.16:0/3713713533 >> v2:172.21.15.156:3300/0 conn(0x7f7e0c072a30 0x7f7e0c165300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e0a7fc640 1 -- 172.21.15.16:0/3713713533 >> v1:172.21.15.156:6789/0 conn(0x7f7e0c0b3750 legacy=0x7f7e0c072320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e0a7fc640 1 -- 172.21.15.16:0/3713713533 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0c15b290 con 0x7f7e0c154ba0 2024-04-24T23:49:44.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.677+0000 7f7e0a7fc640 1 -- 172.21.15.16:0/3713713533 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f7dfc001670 con 0x7f7e0c154ba0 2024-04-24T23:49:44.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0a7fc640 1 -- 172.21.15.16:0/3713713533 >> v1:172.21.15.16:6789/0 conn(0x7f7e0c154ba0 legacy=0x7f7e0c0b5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0a7fc640 1 --2- 172.21.15.16:0/3713713533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7df40039b0 0x7f7df4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0a7fc640 1 --2- 172.21.15.16:0/3713713533 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7df4004800 0x7f7df4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0a7fc640 1 --2- 172.21.15.16:0/3713713533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7df40051c0 0x7f7df400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0a7fc640 1 -- 172.21.15.16:0/3713713533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0c0b5740 con 0x7f7df40051c0 2024-04-24T23:49:44.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0a7fc640 1 -- 172.21.15.16:0/3713713533 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7df400c060 con 0x7f7df40039b0 2024-04-24T23:49:44.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0a7fc640 1 -- 172.21.15.16:0/3713713533 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7df400c260 con 0x7f7df4004800 2024-04-24T23:49:44.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0bfff640 1 --2- 172.21.15.16:0/3713713533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7df40051c0 0x7f7df400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0b7fe640 1 --2- 172.21.15.16:0/3713713533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7df40039b0 0x7f7df4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0affd640 1 --2- 172.21.15.16:0/3713713533 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7df4004800 0x7f7df4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0bfff640 1 -- 172.21.15.16:0/3713713533 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7df4004800 msgr2=0x7f7df4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0bfff640 1 --2- 172.21.15.16:0/3713713533 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7df4004800 0x7f7df4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0bfff640 1 -- 172.21.15.16:0/3713713533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7df40039b0 msgr2=0x7f7df4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0bfff640 1 --2- 172.21.15.16:0/3713713533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7df40039b0 0x7f7df4003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0bfff640 1 -- 172.21.15.16:0/3713713533 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df4003520 con 0x7f7df40051c0 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.678+0000 7f7e0affd640 1 --2- 172.21.15.16:0/3713713533 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7df4004800 0x7f7df4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.679+0000 7f7e0bfff640 1 -- 172.21.15.16:0/3713713533 reap_dead start 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.679+0000 7f7e0bfff640 1 --2- 172.21.15.16:0/3713713533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7df40051c0 0x7f7df400b9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7e04009fc0 tx=0x7f7df4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9433358e0b37889b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.679+0000 7f7e0a7fc640 1 -- 172.21.15.16:0/3713713533 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04035020 con 0x7f7df40051c0 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.679+0000 7f7e0a7fc640 1 -- 172.21.15.16:0/3713713533 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e04003bd0 con 0x7f7df40051c0 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.679+0000 7f7e0a7fc640 1 -- 172.21.15.16:0/3713713533 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04009c10 con 0x7f7df40051c0 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.679+0000 7f7e120a5640 1 -- 172.21.15.16:0/3713713533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7df40051c0 msgr2=0x7f7df400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.679+0000 7f7e120a5640 1 --2- 172.21.15.16:0/3713713533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7df40051c0 0x7f7df400b9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7e04009fc0 tx=0x7f7df4005830 comp rx=0 tx=0).stop 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.679+0000 7f7e120a5640 1 -- 172.21.15.16:0/3713713533 shutdown_connections 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.679+0000 7f7e120a5640 1 --2- 172.21.15.16:0/3713713533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7df40051c0 0x7f7df400b9c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.679+0000 7f7e120a5640 1 -- 172.21.15.16:0/3713713533 >> 172.21.15.16:0/3713713533 conn(0x7f7e0c0cfc50 msgr2=0x7f7e0c0d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.679+0000 7f7e120a5640 1 -- 172.21.15.16:0/3713713533 shutdown_connections 2024-04-24T23:49:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.680+0000 7f7e120a5640 1 -- 172.21.15.16:0/3713713533 wait complete. 2024-04-24T23:49:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.681+0000 7f7e120a5640 1 Processor -- start 2024-04-24T23:49:44.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.681+0000 7f7e120a5640 1 -- start start 2024-04-24T23:49:44.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e120a5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7df40051c0 0x7f7e0c139430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e120a5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e0c14c4b0 0x7f7e0c14c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e120a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e0c14a8f0 0x7f7e0c14ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e120a5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0c15ba90 con 0x7f7e0c14a8f0 2024-04-24T23:49:44.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e120a5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0c15a980 con 0x7f7df40051c0 2024-04-24T23:49:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e120a5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7e0c15bc10 con 0x7f7e0c14c4b0 2024-04-24T23:49:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e0bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e0c14a8f0 0x7f7e0c14ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e0bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e0c14a8f0 0x7f7e0c14ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59906/0 (socket says 172.21.15.16:59906) 2024-04-24T23:49:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e0bfff640 1 -- 172.21.15.16:0/662370567 learned_addr learned my addr 172.21.15.16:0/662370567 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e0b7fe640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7df40051c0 0x7f7e0c139430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e0affd640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e0c14c4b0 0x7f7e0c14c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e0bfff640 1 -- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e0c14c4b0 msgr2=0x7f7e0c14c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e0bfff640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e0c14c4b0 0x7f7e0c14c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e0bfff640 1 -- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7df40051c0 msgr2=0x7f7e0c139430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e0bfff640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7df40051c0 0x7f7e0c139430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e0bfff640 1 -- 172.21.15.16:0/662370567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e0c139970 con 0x7f7e0c14a8f0 2024-04-24T23:49:44.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e0affd640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e0c14c4b0 0x7f7e0c14c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e0bfff640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e0c14a8f0 0x7f7e0c14ada0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400e230 tx=0x7f7e04009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7df3fff640 1 -- 172.21.15.16:0/662370567 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e04035030 con 0x7f7e0c14a8f0 2024-04-24T23:49:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7df3fff640 1 -- 172.21.15.16:0/662370567 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e04043070 con 0x7f7e0c14a8f0 2024-04-24T23:49:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7df3fff640 1 -- 172.21.15.16:0/662370567 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0402d070 con 0x7f7e0c14a8f0 2024-04-24T23:49:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e120a5640 1 -- 172.21.15.16:0/662370567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e0c14b5c0 con 0x7f7e0c14a8f0 2024-04-24T23:49:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.682+0000 7f7e120a5640 1 -- 172.21.15.16:0/662370567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e0c14bb80 con 0x7f7e0c14a8f0 2024-04-24T23:49:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.683+0000 7f7e120a5640 1 -- 172.21.15.16:0/662370567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e0c068f90 con 0x7f7e0c14a8f0 2024-04-24T23:49:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.684+0000 7f7df3fff640 1 -- 172.21.15.16:0/662370567 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7e04005d10 con 0x7f7e0c14a8f0 2024-04-24T23:49:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.684+0000 7f7df3fff640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7df806be60 0x7f7df806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.684+0000 7f7df3fff640 1 -- 172.21.15.16:0/662370567 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7e040abf20 con 0x7f7e0c14a8f0 2024-04-24T23:49:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.685+0000 7f7e0b7fe640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7df806be60 0x7f7df806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.685+0000 7f7e0b7fe640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7df806be60 0x7f7df806e320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c072710 tx=0x7f7dfc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:44.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.689+0000 7f7df3fff640 1 -- 172.21.15.16:0/662370567 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7e0c139970 con 0x7f7e0c14a8f0 2024-04-24T23:49:44.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.869+0000 7f7e120a5640 1 -- 172.21.15.16:0/662370567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7e0c0b4be0 con 0x7f7e0c14a8f0 2024-04-24T23:49:44.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.870+0000 7f7df3fff640 1 -- 172.21.15.16:0/662370567 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f7e0407a4d0 con 0x7f7e0c14a8f0 2024-04-24T23:49:44.869 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:44.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.871+0000 7f7e120a5640 1 -- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7df806be60 msgr2=0x7f7df806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.871+0000 7f7e120a5640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7df806be60 0x7f7df806e320 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c072710 tx=0x7f7dfc009040 comp rx=0 tx=0).stop 2024-04-24T23:49:44.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.871+0000 7f7e120a5640 1 -- 172.21.15.16:0/662370567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e0c14a8f0 msgr2=0x7f7e0c14ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.871+0000 7f7e120a5640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e0c14a8f0 0x7f7e0c14ada0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f7e0400e230 tx=0x7f7e04009750 comp rx=0 tx=0).stop 2024-04-24T23:49:44.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.871+0000 7f7e120a5640 1 -- 172.21.15.16:0/662370567 shutdown_connections 2024-04-24T23:49:44.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.871+0000 7f7e120a5640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7df806be60 0x7f7df806e320 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.871+0000 7f7e120a5640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e0c14a8f0 0x7f7e0c14ada0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.871+0000 7f7e120a5640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e0c14c4b0 0x7f7e0c14c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.871+0000 7f7e120a5640 1 --2- 172.21.15.16:0/662370567 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7df40051c0 0x7f7e0c139430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.871+0000 7f7e120a5640 1 -- 172.21.15.16:0/662370567 >> 172.21.15.16:0/662370567 conn(0x7f7e0c0cfc50 msgr2=0x7f7e0c06dd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:44.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.872+0000 7f7e120a5640 1 -- 172.21.15.16:0/662370567 shutdown_connections 2024-04-24T23:49:44.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.872+0000 7f7e120a5640 1 -- 172.21.15.16:0/662370567 wait complete. 2024-04-24T23:49:44.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:44.986 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.986+0000 7f4a171a2640 1 Processor -- start 2024-04-24T23:49:44.987 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.986+0000 7f4a171a2640 1 -- start start 2024-04-24T23:49:44.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.987+0000 7f4a171a2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f4a1015dfd0 0x7f4a1006b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.987+0000 7f4a171a2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a100b8060 0x7f4a1006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.987+0000 7f4a171a2640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a100d24d0 con 0x7f4a1006bc40 2024-04-24T23:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.987+0000 7f4a171a2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f4a100d27b0 con 0x7f4a1015dfd0 2024-04-24T23:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.987+0000 7f4a171a2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f4a100d2a90 con 0x7f4a100b8060 2024-04-24T23:49:44.990 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.987+0000 7f4a0ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4a1006bc40 0x7f4a100cefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52354/0 (socket says 172.21.15.16:52354) 2024-04-24T23:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.987+0000 7f4a0ffff640 1 -- 172.21.15.16:0/91998147 learned_addr learned my addr 172.21.15.16:0/91998147 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.988+0000 7f4a14f3e640 1 --2- 172.21.15.16:0/91998147 >> v2:172.21.15.156:3300/0 conn(0x7f4a1015dfd0 0x7f4a1006b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.988+0000 7f4a1573f640 1 --2- 172.21.15.16:0/91998147 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a100b8060 0x7f4a1006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.988+0000 7f4a0f7fe640 1 -- 172.21.15.16:0/91998147 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1896719101 0 0) 0x7f4a100d24d0 con 0x7f4a1006bc40 2024-04-24T23:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.988+0000 7f4a0f7fe640 1 -- 172.21.15.16:0/91998147 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49f8003520 con 0x7f4a1006bc40 2024-04-24T23:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.988+0000 7f4a14f3e640 1 -- 172.21.15.16:0/91998147 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a100b8060 msgr2=0x7f4a1006c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.988+0000 7f4a14f3e640 1 --2- 172.21.15.16:0/91998147 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a100b8060 0x7f4a1006c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.988+0000 7f4a14f3e640 1 -- 172.21.15.16:0/91998147 >> v1:172.21.15.16:6789/0 conn(0x7f4a1006bc40 legacy=0x7f4a100cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.988+0000 7f4a14f3e640 1 -- 172.21.15.16:0/91998147 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a100d2d70 con 0x7f4a1015dfd0 2024-04-24T23:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.989+0000 7f4a14f3e640 1 --2- 172.21.15.16:0/91998147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a1015dfd0 0x7f4a1006b2f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4a000097a0 tx=0x7f4a0002a100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bab23a032b375c35 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:44.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.989+0000 7f4a0f7fe640 1 -- 172.21.15.16:0/91998147 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a00037070 con 0x7f4a1015dfd0 2024-04-24T23:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.989+0000 7f4a0f7fe640 1 -- 172.21.15.16:0/91998147 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a0002ae50 con 0x7f4a1015dfd0 2024-04-24T23:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.989+0000 7f4a0f7fe640 1 -- 172.21.15.16:0/91998147 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a00032660 con 0x7f4a1015dfd0 2024-04-24T23:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.989+0000 7f4a171a2640 1 -- 172.21.15.16:0/91998147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a1015dfd0 msgr2=0x7f4a1006b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.989+0000 7f4a171a2640 1 --2- 172.21.15.16:0/91998147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a1015dfd0 0x7f4a1006b2f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4a000097a0 tx=0x7f4a0002a100 comp rx=0 tx=0).stop 2024-04-24T23:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.989+0000 7f4a171a2640 1 -- 172.21.15.16:0/91998147 shutdown_connections 2024-04-24T23:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.989+0000 7f4a171a2640 1 --2- 172.21.15.16:0/91998147 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a100b8060 0x7f4a1006c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.989+0000 7f4a171a2640 1 --2- 172.21.15.16:0/91998147 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a1015dfd0 0x7f4a1006b2f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.989+0000 7f4a171a2640 1 -- 172.21.15.16:0/91998147 >> 172.21.15.16:0/91998147 conn(0x7f4a100d3dc0 msgr2=0x7f4a100d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.989+0000 7f4a171a2640 1 -- 172.21.15.16:0/91998147 shutdown_connections 2024-04-24T23:49:44.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.989+0000 7f4a171a2640 1 -- 172.21.15.16:0/91998147 wait complete. 2024-04-24T23:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.991+0000 7f4a171a2640 1 Processor -- start 2024-04-24T23:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.991+0000 7f4a171a2640 1 -- start start 2024-04-24T23:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.991+0000 7f4a171a2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a1006bc40 0x7f4a100cc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.991+0000 7f4a171a2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a100b8060 0x7f4a100cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.991+0000 7f4a171a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a1015dfd0 0x7f4a10159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.991+0000 7f4a171a2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4a1006b830 con 0x7f4a1015dfd0 2024-04-24T23:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.991+0000 7f4a171a2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4a100cead0 con 0x7f4a1006bc40 2024-04-24T23:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.991+0000 7f4a171a2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f4a1006b9b0 con 0x7f4a100b8060 2024-04-24T23:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a1573f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a1015dfd0 0x7f4a10159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a1573f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a1015dfd0 0x7f4a10159c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59920/0 (socket says 172.21.15.16:59920) 2024-04-24T23:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a1573f640 1 -- 172.21.15.16:0/1465468283 learned_addr learned my addr 172.21.15.16:0/1465468283 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a14f3e640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a1006bc40 0x7f4a100cc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a0ffff640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a100b8060 0x7f4a100cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a1573f640 1 -- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a100b8060 msgr2=0x7f4a100cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a1573f640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a100b8060 0x7f4a100cd6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a1573f640 1 -- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a1006bc40 msgr2=0x7f4a100cc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:44.997 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a1573f640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a1006bc40 0x7f4a100cc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a1573f640 1 -- 172.21.15.16:0/1465468283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a10158e80 con 0x7f4a1015dfd0 2024-04-24T23:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a0ffff640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a100b8060 0x7f4a100cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a1573f640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a1015dfd0 0x7f4a10159c40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4a0800efc0 tx=0x7f4a0800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a0d7fa640 1 -- 172.21.15.16:0/1465468283 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a08018020 con 0x7f4a1015dfd0 2024-04-24T23:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a0d7fa640 1 -- 172.21.15.16:0/1465468283 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a08014480 con 0x7f4a1015dfd0 2024-04-24T23:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a171a2640 1 -- 172.21.15.16:0/1465468283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a10159160 con 0x7f4a1015dfd0 2024-04-24T23:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a0d7fa640 1 -- 172.21.15.16:0/1465468283 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a08014870 con 0x7f4a1015dfd0 2024-04-24T23:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.992+0000 7f4a171a2640 1 -- 172.21.15.16:0/1465468283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a10159720 con 0x7f4a1015dfd0 2024-04-24T23:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.993+0000 7f4a171a2640 1 -- 172.21.15.16:0/1465468283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a10068f90 con 0x7f4a1015dfd0 2024-04-24T23:49:44.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.994+0000 7f4a0d7fa640 1 -- 172.21.15.16:0/1465468283 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4a08005d10 con 0x7f4a1015dfd0 2024-04-24T23:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.994+0000 7f4a0d7fa640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f49fc06be30 0x7f49fc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.994+0000 7f4a0d7fa640 1 -- 172.21.15.16:0/1465468283 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f4a0808bfc0 con 0x7f4a1015dfd0 2024-04-24T23:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.994+0000 7f4a14f3e640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f49fc06be30 0x7f49fc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.995+0000 7f4a14f3e640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f49fc06be30 0x7f49fc06e2f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4a00034000 tx=0x7f4a0002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:44.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:44.998+0000 7f4a0d7fa640 1 -- 172.21.15.16:0/1465468283 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4a08022030 con 0x7f4a1015dfd0 2024-04-24T23:49:45.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.180+0000 7f4a171a2640 1 -- 172.21.15.16:0/1465468283 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f4a101598f0 con 0x7f4a1015dfd0 2024-04-24T23:49:45.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.180+0000 7f4a0d7fa640 1 -- 172.21.15.16:0/1465468283 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f4a08010040 con 0x7f4a1015dfd0 2024-04-24T23:49:45.180 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:45.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.182+0000 7f4a171a2640 1 -- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f49fc06be30 msgr2=0x7f49fc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.182+0000 7f4a171a2640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f49fc06be30 0x7f49fc06e2f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4a00034000 tx=0x7f4a0002e000 comp rx=0 tx=0).stop 2024-04-24T23:49:45.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.182+0000 7f4a171a2640 1 -- 172.21.15.16:0/1465468283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a1015dfd0 msgr2=0x7f4a10159c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.182+0000 7f4a171a2640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a1015dfd0 0x7f4a10159c40 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4a0800efc0 tx=0x7f4a0800c490 comp rx=0 tx=0).stop 2024-04-24T23:49:45.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.182+0000 7f4a171a2640 1 -- 172.21.15.16:0/1465468283 shutdown_connections 2024-04-24T23:49:45.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.182+0000 7f4a171a2640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f49fc06be30 0x7f49fc06e2f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.182+0000 7f4a171a2640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4a1015dfd0 0x7f4a10159c40 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.182+0000 7f4a171a2640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4a100b8060 0x7f4a100cd6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.182+0000 7f4a171a2640 1 --2- 172.21.15.16:0/1465468283 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a1006bc40 0x7f4a100cc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.182+0000 7f4a171a2640 1 -- 172.21.15.16:0/1465468283 >> 172.21.15.16:0/1465468283 conn(0x7f4a100d3dc0 msgr2=0x7f4a100b49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:45.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.183+0000 7f4a171a2640 1 -- 172.21.15.16:0/1465468283 shutdown_connections 2024-04-24T23:49:45.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.183+0000 7f4a171a2640 1 -- 172.21.15.16:0/1465468283 wait complete. 2024-04-24T23:49:45.195 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.300+0000 7fb20f19e640 1 Processor -- start 2024-04-24T23:49:45.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.300+0000 7fb20f19e640 1 -- start start 2024-04-24T23:49:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.301+0000 7fb20f19e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb2080d0cf0 0x7fb208161960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.301+0000 7fb20f19e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2081470b0 0x7fb2080cb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.301+0000 7fb20f19e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb208161ff0 con 0x7fb2081469a0 2024-04-24T23:49:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.301+0000 7fb20f19e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fb208162890 con 0x7fb2080d0cf0 2024-04-24T23:49:45.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.301+0000 7fb20f19e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb2080cb2d0 con 0x7fb2081470b0 2024-04-24T23:49:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.302+0000 7fb20e99d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2081470b0 0x7fb2080cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.302+0000 7fb20d99b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb2080d0cf0 0x7fb208161960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.302+0000 7fb20e99d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2081470b0 0x7fb2080cb760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:53616/0 (socket says 172.21.15.16:53616) 2024-04-24T23:49:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.302+0000 7fb20e99d640 1 -- 172.21.15.16:0/3754826908 learned_addr learned my addr 172.21.15.16:0/3754826908 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.302+0000 7fb20e19c640 1 --1- 172.21.15.16:0/3754826908 >> v1:172.21.15.156:6789/0 conn(0x7fb2081469a0 0x7fb2080d05e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:53034/0 (socket says 172.21.15.16:53034) 2024-04-24T23:49:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.302+0000 7fb20e99d640 1 -- 172.21.15.16:0/3754826908 >> v2:172.21.15.156:3300/0 conn(0x7fb2080d0cf0 msgr2=0x7fb208161960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.302+0000 7fb20e99d640 1 --2- 172.21.15.16:0/3754826908 >> v2:172.21.15.156:3300/0 conn(0x7fb2080d0cf0 0x7fb208161960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.302+0000 7fb20e99d640 1 -- 172.21.15.16:0/3754826908 >> v1:172.21.15.156:6789/0 conn(0x7fb2081469a0 legacy=0x7fb2080d05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.302+0000 7fb20e99d640 1 -- 172.21.15.16:0/3754826908 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2081622d0 con 0x7fb2081470b0 2024-04-24T23:49:45.303 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.302+0000 7fb20d99b640 1 --2- 172.21.15.16:0/3754826908 >> v2:172.21.15.156:3300/0 conn(0x7fb2080d0cf0 0x7fb208161960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:45.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.303+0000 7fb20e99d640 1 --2- 172.21.15.16:0/3754826908 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2081470b0 0x7fb2080cb760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8009870 tx=0x7fb1f802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b092fe8942c0b28f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:45.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.303+0000 7fb20d19a640 1 -- 172.21.15.16:0/3754826908 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8037070 con 0x7fb2081470b0 2024-04-24T23:49:45.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.303+0000 7fb20d19a640 1 -- 172.21.15.16:0/3754826908 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1f8032440 con 0x7fb2081470b0 2024-04-24T23:49:45.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.303+0000 7fb20d19a640 1 -- 172.21.15.16:0/3754826908 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8032810 con 0x7fb2081470b0 2024-04-24T23:49:45.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.303+0000 7fb20f19e640 1 -- 172.21.15.16:0/3754826908 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2081470b0 msgr2=0x7fb2080cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.303+0000 7fb20f19e640 1 --2- 172.21.15.16:0/3754826908 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2081470b0 0x7fb2080cb760 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8009870 tx=0x7fb1f802a1c0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.304+0000 7fb20f19e640 1 -- 172.21.15.16:0/3754826908 shutdown_connections 2024-04-24T23:49:45.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.304+0000 7fb20f19e640 1 --2- 172.21.15.16:0/3754826908 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2081470b0 0x7fb2080cb760 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.304+0000 7fb20f19e640 1 --2- 172.21.15.16:0/3754826908 >> v2:172.21.15.156:3300/0 conn(0x7fb2080d0cf0 0x7fb208161960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.304+0000 7fb20f19e640 1 -- 172.21.15.16:0/3754826908 >> 172.21.15.16:0/3754826908 conn(0x7fb20806b2c0 msgr2=0x7fb20806b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.304+0000 7fb20f19e640 1 -- 172.21.15.16:0/3754826908 shutdown_connections 2024-04-24T23:49:45.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.304+0000 7fb20f19e640 1 -- 172.21.15.16:0/3754826908 wait complete. 2024-04-24T23:49:45.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.305+0000 7fb20f19e640 1 Processor -- start 2024-04-24T23:49:45.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.305+0000 7fb20f19e640 1 -- start start 2024-04-24T23:49:45.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20f19e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2080d0cf0 0x7fb20815d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20f19e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2081469a0 0x7fb2081449f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20f19e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2081470b0 0x7fb208144f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20f19e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb208146d80 con 0x7fb2080d0cf0 2024-04-24T23:49:45.307 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20f19e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb2081478a0 con 0x7fb2081469a0 2024-04-24T23:49:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20f19e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb20815d800 con 0x7fb2081470b0 2024-04-24T23:49:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20e19c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2080d0cf0 0x7fb20815d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20e19c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2080d0cf0 0x7fb20815d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59932/0 (socket says 172.21.15.16:59932) 2024-04-24T23:49:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20e19c640 1 -- 172.21.15.16:0/1017691169 learned_addr learned my addr 172.21.15.16:0/1017691169 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20e99d640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2081470b0 0x7fb208144f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20d99b640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2081469a0 0x7fb2081449f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20e19c640 1 -- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2081470b0 msgr2=0x7fb208144f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20e19c640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2081470b0 0x7fb208144f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20e19c640 1 -- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2081469a0 msgr2=0x7fb2081449f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20e19c640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2081469a0 0x7fb2081449f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20e19c640 1 -- 172.21.15.16:0/1017691169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb20806daf0 con 0x7fb2080d0cf0 2024-04-24T23:49:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20e99d640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2081470b0 0x7fb208144f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20d99b640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2081469a0 0x7fb2081449f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb20e19c640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2080d0cf0 0x7fb20815d2c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb20000c780 tx=0x7fb20000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb206ffd640 1 -- 172.21.15.16:0/1017691169 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb200017070 con 0x7fb2080d0cf0 2024-04-24T23:49:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.306+0000 7fb206ffd640 1 -- 172.21.15.16:0/1017691169 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb20000a640 con 0x7fb2080d0cf0 2024-04-24T23:49:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.307+0000 7fb20f19e640 1 -- 172.21.15.16:0/1017691169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb20806ddd0 con 0x7fb2080d0cf0 2024-04-24T23:49:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.307+0000 7fb206ffd640 1 -- 172.21.15.16:0/1017691169 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20000aab0 con 0x7fb2080d0cf0 2024-04-24T23:49:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.307+0000 7fb20f19e640 1 -- 172.21.15.16:0/1017691169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb20806e390 con 0x7fb2080d0cf0 2024-04-24T23:49:45.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.307+0000 7fb20f19e640 1 -- 172.21.15.16:0/1017691169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb20804f500 con 0x7fb2080d0cf0 2024-04-24T23:49:45.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.308+0000 7fb206ffd640 1 -- 172.21.15.16:0/1017691169 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb20001a070 con 0x7fb2080d0cf0 2024-04-24T23:49:45.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.308+0000 7fb206ffd640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb1f406be30 0x7fb1f406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.309+0000 7fb206ffd640 1 -- 172.21.15.16:0/1017691169 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb20008a820 con 0x7fb2080d0cf0 2024-04-24T23:49:45.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.309+0000 7fb20d99b640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb1f406be30 0x7fb1f406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.310 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.309+0000 7fb20d99b640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb1f406be30 0x7fb1f406e2f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb2080cca80 tx=0x7fb1fc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:45.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.313+0000 7fb206ffd640 1 -- 172.21.15.16:0/1017691169 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb200058e60 con 0x7fb2080d0cf0 2024-04-24T23:49:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.497+0000 7fb20f19e640 1 -- 172.21.15.16:0/1017691169 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb20813ee40 con 0x7fb2080d0cf0 2024-04-24T23:49:45.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.498+0000 7fb206ffd640 1 -- 172.21.15.16:0/1017691169 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fb200058760 con 0x7fb2080d0cf0 2024-04-24T23:49:45.497 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.500+0000 7fb20f19e640 1 -- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb1f406be30 msgr2=0x7fb1f406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.500+0000 7fb20f19e640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb1f406be30 0x7fb1f406e2f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb2080cca80 tx=0x7fb1fc008040 comp rx=0 tx=0).stop 2024-04-24T23:49:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.500+0000 7fb20f19e640 1 -- 172.21.15.16:0/1017691169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2080d0cf0 msgr2=0x7fb20815d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.500+0000 7fb20f19e640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2080d0cf0 0x7fb20815d2c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb20000c780 tx=0x7fb20000cc40 comp rx=0 tx=0).stop 2024-04-24T23:49:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.500+0000 7fb20f19e640 1 -- 172.21.15.16:0/1017691169 shutdown_connections 2024-04-24T23:49:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.500+0000 7fb20f19e640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb1f406be30 0x7fb1f406e2f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.500+0000 7fb20f19e640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb2081470b0 0x7fb208144f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.500+0000 7fb20f19e640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb2081469a0 0x7fb2081449f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.500+0000 7fb20f19e640 1 --2- 172.21.15.16:0/1017691169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2080d0cf0 0x7fb20815d2c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.500+0000 7fb20f19e640 1 -- 172.21.15.16:0/1017691169 >> 172.21.15.16:0/1017691169 conn(0x7fb20806b2c0 msgr2=0x7fb2080ce3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.500+0000 7fb20f19e640 1 -- 172.21.15.16:0/1017691169 shutdown_connections 2024-04-24T23:49:45.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.500+0000 7fb20f19e640 1 -- 172.21.15.16:0/1017691169 wait complete. 2024-04-24T23:49:45.532 INFO:tasks.cephfs_test_runner:Starting test: test_recover_fs_after_fsmap_removal (tasks.cephfs.test_recovery_fs.TestFSRecovery) 2024-04-24T23:49:45.533 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.634+0000 7f15fc81c640 1 Processor -- start 2024-04-24T23:49:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.634+0000 7f15fc81c640 1 -- start start 2024-04-24T23:49:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.635+0000 7f15fc81c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15f815e020 0x7f15f80cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.635+0000 7f15fc81c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f15f80d1fa0 0x7f15f80d2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.635+0000 7f15fc81c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15f80b3390 con 0x7f15f80b2bb0 2024-04-24T23:49:45.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.635+0000 7f15fc81c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f15f806bfc0 con 0x7f15f80d1fa0 2024-04-24T23:49:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.635+0000 7f15fc81c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f15f806c2a0 con 0x7f15f815e020 2024-04-24T23:49:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.635+0000 7f15f5d9b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f15f80b2bb0 0x7f15f80cf7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52370/0 (socket says 172.21.15.16:52370) 2024-04-24T23:49:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.635+0000 7f15f5d9b640 1 -- 172.21.15.16:0/808312239 learned_addr learned my addr 172.21.15.16:0/808312239 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f6d9d640 1 --2- 172.21.15.16:0/808312239 >> v2:172.21.15.156:3300/0 conn(0x7f15f80d1fa0 0x7f15f80d2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f659c640 1 --2- 172.21.15.16:0/808312239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15f815e020 0x7f15f80cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 -- 172.21.15.16:0/808312239 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2531850847 0 0) 0x7f15f80b3390 con 0x7f15f80b2bb0 2024-04-24T23:49:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 -- 172.21.15.16:0/808312239 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15e4003520 con 0x7f15f80b2bb0 2024-04-24T23:49:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 -- 172.21.15.16:0/808312239 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2255641379 0 0) 0x7f15e4003520 con 0x7f15f80b2bb0 2024-04-24T23:49:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 -- 172.21.15.16:0/808312239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15f815e020 msgr2=0x7f15f80cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 --2- 172.21.15.16:0/808312239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15f815e020 0x7f15f80cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 -- 172.21.15.16:0/808312239 >> v2:172.21.15.156:3300/0 conn(0x7f15f80d1fa0 msgr2=0x7f15f80d2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 --2- 172.21.15.16:0/808312239 >> v2:172.21.15.156:3300/0 conn(0x7f15f80d1fa0 0x7f15f80d2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 -- 172.21.15.16:0/808312239 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f806c580 con 0x7f15f80b2bb0 2024-04-24T23:49:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 -- 172.21.15.16:0/808312239 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f15e0001670 con 0x7f15f80b2bb0 2024-04-24T23:49:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 -- 172.21.15.16:0/808312239 >> v1:172.21.15.16:6789/0 conn(0x7f15f80b2bb0 legacy=0x7f15f80cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 --2- 172.21.15.16:0/808312239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e40039b0 0x7f15e4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 --2- 172.21.15.16:0/808312239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15f815e020 0x7f15e4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 --2- 172.21.15.16:0/808312239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e4004d40 0x7f15e40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 -- 172.21.15.16:0/808312239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15f806bfc0 con 0x7f15e40039b0 2024-04-24T23:49:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 -- 172.21.15.16:0/808312239 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f15f806c2a0 con 0x7f15e4004d40 2024-04-24T23:49:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f559a640 1 -- 172.21.15.16:0/808312239 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f15e400b9c0 con 0x7f15f815e020 2024-04-24T23:49:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f659c640 1 --2- 172.21.15.16:0/808312239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e40039b0 0x7f15e4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f5d9b640 1 --2- 172.21.15.16:0/808312239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15f815e020 0x7f15e4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f6d9d640 1 --2- 172.21.15.16:0/808312239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e4004d40 0x7f15e40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f659c640 1 -- 172.21.15.16:0/808312239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15f815e020 msgr2=0x7f15e4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f659c640 1 --2- 172.21.15.16:0/808312239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15f815e020 0x7f15e4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f659c640 1 -- 172.21.15.16:0/808312239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e4004d40 msgr2=0x7f15e40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f659c640 1 --2- 172.21.15.16:0/808312239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e4004d40 0x7f15e40051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f659c640 1 -- 172.21.15.16:0/808312239 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15e4003520 con 0x7f15e40039b0 2024-04-24T23:49:45.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.636+0000 7f15f5d9b640 1 --2- 172.21.15.16:0/808312239 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15f815e020 0x7f15e4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.637+0000 7f15f659c640 1 --2- 172.21.15.16:0/808312239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e40039b0 0x7f15e4003dd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009ac0 tx=0x7f15e4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d69b72a8149e892f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.637+0000 7f15f559a640 1 -- 172.21.15.16:0/808312239 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f0036070 con 0x7f15e40039b0 2024-04-24T23:49:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.637+0000 7f15f559a640 1 -- 172.21.15.16:0/808312239 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15f0031400 con 0x7f15e40039b0 2024-04-24T23:49:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.637+0000 7f15f559a640 1 -- 172.21.15.16:0/808312239 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15f0009690 con 0x7f15e40039b0 2024-04-24T23:49:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.638+0000 7f15fc81c640 1 -- 172.21.15.16:0/808312239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e40039b0 msgr2=0x7f15e4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.638+0000 7f15fc81c640 1 --2- 172.21.15.16:0/808312239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15e40039b0 0x7f15e4003dd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f15f0009ac0 tx=0x7f15e4005830 comp rx=0 tx=0).stop 2024-04-24T23:49:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.638+0000 7f15f6d9d640 1 -- 172.21.15.16:0/808312239 reap_dead start 2024-04-24T23:49:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.638+0000 7f15fc81c640 1 -- 172.21.15.16:0/808312239 shutdown_connections 2024-04-24T23:49:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.638+0000 7f15fc81c640 1 -- 172.21.15.16:0/808312239 >> 172.21.15.16:0/808312239 conn(0x7f15f80d3e10 msgr2=0x7f15f80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.638+0000 7f15fc81c640 1 -- 172.21.15.16:0/808312239 shutdown_connections 2024-04-24T23:49:45.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.638+0000 7f15fc81c640 1 -- 172.21.15.16:0/808312239 wait complete. 2024-04-24T23:49:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.639+0000 7f15fc81c640 1 Processor -- start 2024-04-24T23:49:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.639+0000 7f15fc81c640 1 -- start start 2024-04-24T23:49:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.640+0000 7f15fc81c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e40039b0 0x7f15f815b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.640+0000 7f15fc81c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15f80b8920 0x7f15f80b8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.640+0000 7f15fc81c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15f80ca160 0x7f15f80ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.640+0000 7f15fc81c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f15f815e400 con 0x7f15f80b8920 2024-04-24T23:49:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.640+0000 7f15fc81c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f15f80d1a80 con 0x7f15e40039b0 2024-04-24T23:49:45.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.640+0000 7f15fc81c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f15f80cfec0 con 0x7f15f80ca160 2024-04-24T23:49:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.640+0000 7f15f5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15f80b8920 0x7f15f80b8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.640+0000 7f15f5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15f80b8920 0x7f15f80b8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59954/0 (socket says 172.21.15.16:59954) 2024-04-24T23:49:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.640+0000 7f15f5d9b640 1 -- 172.21.15.16:0/1062030422 learned_addr learned my addr 172.21.15.16:0/1062030422 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.640+0000 7f15f6d9d640 1 --2- 172.21.15.16:0/1062030422 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15f80ca160 0x7f15f80ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.640+0000 7f15f659c640 1 --2- 172.21.15.16:0/1062030422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e40039b0 0x7f15f815b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.641+0000 7f15f5d9b640 1 -- 172.21.15.16:0/1062030422 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15f80ca160 msgr2=0x7f15f80ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.641+0000 7f15f5d9b640 1 --2- 172.21.15.16:0/1062030422 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15f80ca160 0x7f15f80ca5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.641+0000 7f15f5d9b640 1 -- 172.21.15.16:0/1062030422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e40039b0 msgr2=0x7f15f815b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.641+0000 7f15f5d9b640 1 --2- 172.21.15.16:0/1062030422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e40039b0 0x7f15f815b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.641+0000 7f15f5d9b640 1 -- 172.21.15.16:0/1062030422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15f815a8a0 con 0x7f15f80b8920 2024-04-24T23:49:45.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.641+0000 7f15f5d9b640 1 --2- 172.21.15.16:0/1062030422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15f80b8920 0x7f15f80b8da0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f15e000e8d0 tx=0x7f15e000ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.641+0000 7f15eeffd640 1 -- 172.21.15.16:0/1062030422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e000c850 con 0x7f15f80b8920 2024-04-24T23:49:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.641+0000 7f15eeffd640 1 -- 172.21.15.16:0/1062030422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15e0018070 con 0x7f15f80b8920 2024-04-24T23:49:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.641+0000 7f15fc81c640 1 -- 172.21.15.16:0/1062030422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15f815ab80 con 0x7f15f80b8920 2024-04-24T23:49:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.641+0000 7f15eeffd640 1 -- 172.21.15.16:0/1062030422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15e000cc10 con 0x7f15f80b8920 2024-04-24T23:49:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.641+0000 7f15fc81c640 1 -- 172.21.15.16:0/1062030422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15f81554e0 con 0x7f15f80b8920 2024-04-24T23:49:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.642+0000 7f15fc81c640 1 -- 172.21.15.16:0/1062030422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15f806dfe0 con 0x7f15f80b8920 2024-04-24T23:49:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.643+0000 7f15eeffd640 1 -- 172.21.15.16:0/1062030422 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f15e001d460 con 0x7f15f80b8920 2024-04-24T23:49:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.643+0000 7f15eeffd640 1 --2- 172.21.15.16:0/1062030422 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f15d806be30 0x7f15d806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.643+0000 7f15eeffd640 1 -- 172.21.15.16:0/1062030422 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f15e008b130 con 0x7f15f80b8920 2024-04-24T23:49:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.644+0000 7f15f659c640 1 --2- 172.21.15.16:0/1062030422 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f15d806be30 0x7f15d806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.644+0000 7f15f659c640 1 --2- 172.21.15.16:0/1062030422 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f15d806be30 0x7f15d806e2f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f15f0033000 tx=0x7f15f0008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:45.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.648+0000 7f15eeffd640 1 -- 172.21.15.16:0/1062030422 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f15e0059090 con 0x7f15f80b8920 2024-04-24T23:49:45.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.833+0000 7f15fc81c640 1 -- 172.21.15.16:0/1062030422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15f804f5e0 con 0x7f15f80b8920 2024-04-24T23:49:45.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.834+0000 7f15eeffd640 1 -- 172.21.15.16:0/1062030422 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7f15e005d390 con 0x7f15f80b8920 2024-04-24T23:49:45.834 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:45.834 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:45.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.836+0000 7f15fc81c640 1 -- 172.21.15.16:0/1062030422 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f15d806be30 msgr2=0x7f15d806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.836+0000 7f15fc81c640 1 --2- 172.21.15.16:0/1062030422 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f15d806be30 0x7f15d806e2f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f15f0033000 tx=0x7f15f0008030 comp rx=0 tx=0).stop 2024-04-24T23:49:45.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.836+0000 7f15fc81c640 1 -- 172.21.15.16:0/1062030422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15f80b8920 msgr2=0x7f15f80b8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.836+0000 7f15fc81c640 1 --2- 172.21.15.16:0/1062030422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15f80b8920 0x7f15f80b8da0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f15e000e8d0 tx=0x7f15e000ed90 comp rx=0 tx=0).stop 2024-04-24T23:49:45.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.836+0000 7f15fc81c640 1 -- 172.21.15.16:0/1062030422 shutdown_connections 2024-04-24T23:49:45.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.836+0000 7f15fc81c640 1 --2- 172.21.15.16:0/1062030422 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f15d806be30 0x7f15d806e2f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.836+0000 7f15fc81c640 1 --2- 172.21.15.16:0/1062030422 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f15f80ca160 0x7f15f80ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.836+0000 7f15fc81c640 1 --2- 172.21.15.16:0/1062030422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f15f80b8920 0x7f15f80b8da0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.836+0000 7f15fc81c640 1 --2- 172.21.15.16:0/1062030422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15e40039b0 0x7f15f815b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.836+0000 7f15fc81c640 1 -- 172.21.15.16:0/1062030422 >> 172.21.15.16:0/1062030422 conn(0x7f15f80d3e10 msgr2=0x7f15f80b79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:45.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.836+0000 7f15fc81c640 1 -- 172.21.15.16:0/1062030422 shutdown_connections 2024-04-24T23:49:45.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.836+0000 7f15fc81c640 1 -- 172.21.15.16:0/1062030422 wait complete. 2024-04-24T23:49:45.848 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:45.849 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.954+0000 7fc4403ee640 1 Processor -- start 2024-04-24T23:49:45.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.954+0000 7fc4403ee640 1 -- start start 2024-04-24T23:49:45.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc4403ee640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc4381444c0 0x7fc438071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc4403ee640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc438071fd0 0x7fc4380ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc4403ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4380ce870 0x7fc4380b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc4403ee640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc438072430 con 0x7fc438071fd0 2024-04-24T23:49:45.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc4403ee640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fc4380cf0a0 con 0x7fc4381444c0 2024-04-24T23:49:45.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc4403ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fc43806bbf0 con 0x7fc4380ce870 2024-04-24T23:49:45.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc43d989640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc438071fd0 0x7fc4380ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc43d989640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc438071fd0 0x7fc4380ce330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59968/0 (socket says 172.21.15.16:59968) 2024-04-24T23:49:45.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc43d989640 1 -- 172.21.15.16:0/3662456884 learned_addr learned my addr 172.21.15.16:0/3662456884 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:45.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc43e18a640 1 --2- 172.21.15.16:0/3662456884 >> v2:172.21.15.156:3300/0 conn(0x7fc4381444c0 0x7fc438071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc43e98b640 1 --2- 172.21.15.16:0/3662456884 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4380ce870 0x7fc4380b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc43d989640 1 -- 172.21.15.16:0/3662456884 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4380ce870 msgr2=0x7fc4380b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:45.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc43d989640 1 --2- 172.21.15.16:0/3662456884 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4380ce870 0x7fc4380b3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc43d989640 1 -- 172.21.15.16:0/3662456884 >> v2:172.21.15.156:3300/0 conn(0x7fc4381444c0 msgr2=0x7fc438071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:45.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc43d989640 1 --2- 172.21.15.16:0/3662456884 >> v2:172.21.15.156:3300/0 conn(0x7fc4381444c0 0x7fc438071a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.955+0000 7fc43d989640 1 -- 172.21.15.16:0/3662456884 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc43806be90 con 0x7fc438071fd0 2024-04-24T23:49:45.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.956+0000 7fc43d989640 1 --2- 172.21.15.16:0/3662456884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438071fd0 0x7fc4380ce330 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc4380b5d90 tx=0x7fc42c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd56c8cd23db09e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.956+0000 7fc43d188640 1 -- 172.21.15.16:0/3662456884 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c037070 con 0x7fc438071fd0 2024-04-24T23:49:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.956+0000 7fc43d188640 1 -- 172.21.15.16:0/3662456884 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc42c032440 con 0x7fc438071fd0 2024-04-24T23:49:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.956+0000 7fc43d188640 1 -- 172.21.15.16:0/3662456884 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc42c032810 con 0x7fc438071fd0 2024-04-24T23:49:45.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.957+0000 7fc4403ee640 1 -- 172.21.15.16:0/3662456884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438071fd0 msgr2=0x7fc4380ce330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.957+0000 7fc4403ee640 1 --2- 172.21.15.16:0/3662456884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438071fd0 0x7fc4380ce330 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fc4380b5d90 tx=0x7fc42c02a1e0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.957+0000 7fc4403ee640 1 -- 172.21.15.16:0/3662456884 shutdown_connections 2024-04-24T23:49:45.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.957+0000 7fc4403ee640 1 --2- 172.21.15.16:0/3662456884 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4380ce870 0x7fc4380b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.957+0000 7fc4403ee640 1 --2- 172.21.15.16:0/3662456884 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438071fd0 0x7fc4380ce330 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.957+0000 7fc4403ee640 1 --2- 172.21.15.16:0/3662456884 >> v2:172.21.15.156:3300/0 conn(0x7fc4381444c0 0x7fc438071a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.957+0000 7fc4403ee640 1 -- 172.21.15.16:0/3662456884 >> 172.21.15.16:0/3662456884 conn(0x7fc4380cfc50 msgr2=0x7fc4380d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:45.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.957+0000 7fc4403ee640 1 -- 172.21.15.16:0/3662456884 shutdown_connections 2024-04-24T23:49:45.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.957+0000 7fc4403ee640 1 -- 172.21.15.16:0/3662456884 wait complete. 2024-04-24T23:49:45.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.958+0000 7fc4403ee640 1 Processor -- start 2024-04-24T23:49:45.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.958+0000 7fc4403ee640 1 -- start start 2024-04-24T23:49:45.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc4403ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438071fd0 0x7fc438157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc4403ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4380ce870 0x7fc43813a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc4403ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4381444c0 0x7fc438159620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc4403ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc43806c730 con 0x7fc438071fd0 2024-04-24T23:49:45.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc4403ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc43806c450 con 0x7fc4381444c0 2024-04-24T23:49:45.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc4403ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fc4380ced00 con 0x7fc4380ce870 2024-04-24T23:49:45.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc43e18a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438071fd0 0x7fc438157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc43e18a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438071fd0 0x7fc438157aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59974/0 (socket says 172.21.15.16:59974) 2024-04-24T23:49:45.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc43e18a640 1 -- 172.21.15.16:0/1799196484 learned_addr learned my addr 172.21.15.16:0/1799196484 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:45.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc43e98b640 1 --2- 172.21.15.16:0/1799196484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4381444c0 0x7fc438159620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc43d989640 1 --2- 172.21.15.16:0/1799196484 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4380ce870 0x7fc43813a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc43e18a640 1 -- 172.21.15.16:0/1799196484 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4380ce870 msgr2=0x7fc43813a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc43e18a640 1 --2- 172.21.15.16:0/1799196484 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4380ce870 0x7fc43813a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc43e18a640 1 -- 172.21.15.16:0/1799196484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4381444c0 msgr2=0x7fc438159620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:45.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc43e18a640 1 --2- 172.21.15.16:0/1799196484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4381444c0 0x7fc438159620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:45.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc43e18a640 1 -- 172.21.15.16:0/1799196484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc438156a00 con 0x7fc438071fd0 2024-04-24T23:49:45.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.959+0000 7fc43e18a640 1 --2- 172.21.15.16:0/1799196484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438071fd0 0x7fc438157aa0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc43000d5f0 tx=0x7fc43000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:45.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.960+0000 7fc436ffd640 1 -- 172.21.15.16:0/1799196484 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4300125f0 con 0x7fc438071fd0 2024-04-24T23:49:45.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.960+0000 7fc436ffd640 1 -- 172.21.15.16:0/1799196484 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc430019070 con 0x7fc438071fd0 2024-04-24T23:49:45.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.960+0000 7fc436ffd640 1 -- 172.21.15.16:0/1799196484 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4300129d0 con 0x7fc438071fd0 2024-04-24T23:49:45.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.960+0000 7fc4403ee640 1 -- 172.21.15.16:0/1799196484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc438156ce0 con 0x7fc438071fd0 2024-04-24T23:49:45.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.960+0000 7fc4403ee640 1 -- 172.21.15.16:0/1799196484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4381572a0 con 0x7fc438071fd0 2024-04-24T23:49:45.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.960+0000 7fc4403ee640 1 -- 172.21.15.16:0/1799196484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc438068f90 con 0x7fc438071fd0 2024-04-24T23:49:45.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.961+0000 7fc436ffd640 1 -- 172.21.15.16:0/1799196484 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc430012b70 con 0x7fc438071fd0 2024-04-24T23:49:45.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.961+0000 7fc436ffd640 1 --2- 172.21.15.16:0/1799196484 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc42406be30 0x7fc42406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:45.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.961+0000 7fc436ffd640 1 -- 172.21.15.16:0/1799196484 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc43008b560 con 0x7fc438071fd0 2024-04-24T23:49:45.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.962+0000 7fc43d989640 1 --2- 172.21.15.16:0/1799196484 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc42406be30 0x7fc42406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:45.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.962+0000 7fc43d989640 1 --2- 172.21.15.16:0/1799196484 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc42406be30 0x7fc42406e2f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc42c034000 tx=0x7fc42c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:45.965 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:45.966+0000 7fc436ffd640 1 -- 172.21.15.16:0/1799196484 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc4300594c0 con 0x7fc438071fd0 2024-04-24T23:49:46.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.147+0000 7fc4403ee640 1 -- 172.21.15.16:0/1799196484 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc43815b000 con 0x7fc438071fd0 2024-04-24T23:49:46.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.147+0000 7fc436ffd640 1 -- 172.21.15.16:0/1799196484 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fc43005d7c0 con 0x7fc438071fd0 2024-04-24T23:49:46.148 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:46.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.149+0000 7fc4403ee640 1 -- 172.21.15.16:0/1799196484 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc42406be30 msgr2=0x7fc42406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.149+0000 7fc4403ee640 1 --2- 172.21.15.16:0/1799196484 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc42406be30 0x7fc42406e2f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc42c034000 tx=0x7fc42c02e000 comp rx=0 tx=0).stop 2024-04-24T23:49:46.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.150+0000 7fc4403ee640 1 -- 172.21.15.16:0/1799196484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438071fd0 msgr2=0x7fc438157aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.150+0000 7fc4403ee640 1 --2- 172.21.15.16:0/1799196484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438071fd0 0x7fc438157aa0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc43000d5f0 tx=0x7fc43000dab0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.150+0000 7fc4403ee640 1 -- 172.21.15.16:0/1799196484 shutdown_connections 2024-04-24T23:49:46.150 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.150+0000 7fc4403ee640 1 --2- 172.21.15.16:0/1799196484 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc42406be30 0x7fc42406e2f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.150+0000 7fc4403ee640 1 --2- 172.21.15.16:0/1799196484 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc4381444c0 0x7fc438159620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.150+0000 7fc4403ee640 1 --2- 172.21.15.16:0/1799196484 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc4380ce870 0x7fc43813a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.150+0000 7fc4403ee640 1 --2- 172.21.15.16:0/1799196484 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc438071fd0 0x7fc438157aa0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.150+0000 7fc4403ee640 1 -- 172.21.15.16:0/1799196484 >> 172.21.15.16:0/1799196484 conn(0x7fc4380cfc50 msgr2=0x7fc4380af750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:46.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.150+0000 7fc4403ee640 1 -- 172.21.15.16:0/1799196484 shutdown_connections 2024-04-24T23:49:46.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.150+0000 7fc4403ee640 1 -- 172.21.15.16:0/1799196484 wait complete. 2024-04-24T23:49:46.161 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:46.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.272+0000 7f11a6d7e640 1 Processor -- start 2024-04-24T23:49:46.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.272+0000 7f11a6d7e640 1 -- start start 2024-04-24T23:49:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.273+0000 7f11a6d7e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f11a00af980 0x7f11a006deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.273+0000 7f11a6d7e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11a006d360 0x7f11a0071e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.273+0000 7f11a6d7e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11a006d8d0 con 0x7f11a015dff0 2024-04-24T23:49:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.273+0000 7f11a6d7e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f11a00732d0 con 0x7f11a00af980 2024-04-24T23:49:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.273+0000 7f11a6d7e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f11a006dbb0 con 0x7f11a006d360 2024-04-24T23:49:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.273+0000 7f11a4b1a640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f11a015dff0 0x7f11a00af270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52384/0 (socket says 172.21.15.16:52384) 2024-04-24T23:49:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.273+0000 7f11a4b1a640 1 -- 172.21.15.16:0/1374089680 learned_addr learned my addr 172.21.15.16:0/1374089680 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.273+0000 7f11a531b640 1 --2- 172.21.15.16:0/1374089680 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11a006d360 0x7f11a0071e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.273+0000 7f119ffff640 1 --2- 172.21.15.16:0/1374089680 >> v2:172.21.15.156:3300/0 conn(0x7f11a00af980 0x7f11a006deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.273+0000 7f119f7fe640 1 -- 172.21.15.16:0/1374089680 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3092799342 0 0) 0x7f11a006d8d0 con 0x7f11a015dff0 2024-04-24T23:49:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.273+0000 7f119f7fe640 1 -- 172.21.15.16:0/1374089680 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f118c003520 con 0x7f11a015dff0 2024-04-24T23:49:46.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119f7fe640 1 -- 172.21.15.16:0/1374089680 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1820251750 0 0) 0x7f118c003520 con 0x7f11a015dff0 2024-04-24T23:49:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119f7fe640 1 -- 172.21.15.16:0/1374089680 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11a006d360 msgr2=0x7f11a0071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119f7fe640 1 --2- 172.21.15.16:0/1374089680 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11a006d360 0x7f11a0071e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119f7fe640 1 -- 172.21.15.16:0/1374089680 >> v2:172.21.15.156:3300/0 conn(0x7f11a00af980 msgr2=0x7f11a006deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119f7fe640 1 --2- 172.21.15.16:0/1374089680 >> v2:172.21.15.156:3300/0 conn(0x7f11a00af980 0x7f11a006deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119f7fe640 1 -- 172.21.15.16:0/1374089680 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a0071330 con 0x7f11a015dff0 2024-04-24T23:49:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119f7fe640 1 -- 172.21.15.16:0/1374089680 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f1198001670 con 0x7f11a015dff0 2024-04-24T23:49:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119f7fe640 1 -- 172.21.15.16:0/1374089680 >> v1:172.21.15.16:6789/0 conn(0x7f11a015dff0 legacy=0x7f11a00af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119f7fe640 1 --2- 172.21.15.16:0/1374089680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f118c0039b0 0x7f118c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119f7fe640 1 --2- 172.21.15.16:0/1374089680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f118c004800 0x7f118c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119f7fe640 1 --2- 172.21.15.16:0/1374089680 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11a006d360 0x7f118c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.277 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119f7fe640 1 -- 172.21.15.16:0/1374089680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f11a00732d0 con 0x7f118c004800 2024-04-24T23:49:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119f7fe640 1 -- 172.21.15.16:0/1374089680 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f11a006dbb0 con 0x7f118c0039b0 2024-04-24T23:49:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119f7fe640 1 -- 172.21.15.16:0/1374089680 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f118c00b9c0 con 0x7f11a006d360 2024-04-24T23:49:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119ffff640 1 --2- 172.21.15.16:0/1374089680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f118c004800 0x7f118c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f11a4b1a640 1 --2- 172.21.15.16:0/1374089680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f118c0039b0 0x7f118c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f11a531b640 1 --2- 172.21.15.16:0/1374089680 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11a006d360 0x7f118c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119ffff640 1 -- 172.21.15.16:0/1374089680 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11a006d360 msgr2=0x7f118c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119ffff640 1 --2- 172.21.15.16:0/1374089680 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11a006d360 0x7f118c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119ffff640 1 -- 172.21.15.16:0/1374089680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f118c0039b0 msgr2=0x7f118c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119ffff640 1 --2- 172.21.15.16:0/1374089680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f118c0039b0 0x7f118c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f119ffff640 1 -- 172.21.15.16:0/1374089680 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f118c003520 con 0x7f118c004800 2024-04-24T23:49:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.274+0000 7f11a531b640 1 --2- 172.21.15.16:0/1374089680 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11a006d360 0x7f118c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:46.278 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.275+0000 7f119ffff640 1 --2- 172.21.15.16:0/1374089680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f118c004800 0x7f118c004c80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1190009fc0 tx=0x7f118c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae78ef18518b9cf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.275+0000 7f119f7fe640 1 -- 172.21.15.16:0/1374089680 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1190009c10 con 0x7f118c004800 2024-04-24T23:49:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.275+0000 7f119f7fe640 1 -- 172.21.15.16:0/1374089680 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1190036070 con 0x7f118c004800 2024-04-24T23:49:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.275+0000 7f119f7fe640 1 -- 172.21.15.16:0/1374089680 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119002d070 con 0x7f118c004800 2024-04-24T23:49:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.275+0000 7f11a6d7e640 1 -- 172.21.15.16:0/1374089680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f118c004800 msgr2=0x7f118c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.275+0000 7f11a6d7e640 1 --2- 172.21.15.16:0/1374089680 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f118c004800 0x7f118c004c80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1190009fc0 tx=0x7f118c005830 comp rx=0 tx=0).stop 2024-04-24T23:49:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.275+0000 7f11a531b640 1 -- 172.21.15.16:0/1374089680 reap_dead start 2024-04-24T23:49:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.275+0000 7f11a6d7e640 1 -- 172.21.15.16:0/1374089680 shutdown_connections 2024-04-24T23:49:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.275+0000 7f11a6d7e640 1 -- 172.21.15.16:0/1374089680 >> 172.21.15.16:0/1374089680 conn(0x7f11a00affc0 msgr2=0x7f11a00b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:46.279 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.276+0000 7f11a6d7e640 1 -- 172.21.15.16:0/1374089680 shutdown_connections 2024-04-24T23:49:46.280 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.276+0000 7f11a6d7e640 1 -- 172.21.15.16:0/1374089680 wait complete. 2024-04-24T23:49:46.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.277+0000 7f11a6d7e640 1 Processor -- start 2024-04-24T23:49:46.283 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.277+0000 7f11a6d7e640 1 -- start start 2024-04-24T23:49:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a6d7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f118c004800 0x7f11a0141750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a6d7e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11a0150ee0 0x7f11a015ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a6d7e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11a01f70f0 0x7f11a01f3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a6d7e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f11a00ccca0 con 0x7f118c004800 2024-04-24T23:49:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a6d7e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f11a00718f0 con 0x7f11a0150ee0 2024-04-24T23:49:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a6d7e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f11a00cce20 con 0x7f11a01f70f0 2024-04-24T23:49:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a4b1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f118c004800 0x7f11a0141750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a4b1a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f118c004800 0x7f11a0141750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59992/0 (socket says 172.21.15.16:59992) 2024-04-24T23:49:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a4b1a640 1 -- 172.21.15.16:0/3967772770 learned_addr learned my addr 172.21.15.16:0/3967772770 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f119ffff640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11a0150ee0 0x7f11a015ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a531b640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11a01f70f0 0x7f11a01f3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a4b1a640 1 -- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11a01f70f0 msgr2=0x7f11a01f3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a4b1a640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11a01f70f0 0x7f11a01f3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a4b1a640 1 -- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11a0150ee0 msgr2=0x7f11a015ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a4b1a640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11a0150ee0 0x7f11a015ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a4b1a640 1 -- 172.21.15.16:0/3967772770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11a0161310 con 0x7f118c004800 2024-04-24T23:49:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f119ffff640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11a0150ee0 0x7f11a015ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a4b1a640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f118c004800 0x7f11a0141750 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1198007310 tx=0x7f11980077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f119d7fa640 1 -- 172.21.15.16:0/3967772770 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119800f960 con 0x7f118c004800 2024-04-24T23:49:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f119d7fa640 1 -- 172.21.15.16:0/3967772770 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1198018070 con 0x7f118c004800 2024-04-24T23:49:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f11a6d7e640 1 -- 172.21.15.16:0/3967772770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11a01f7a60 con 0x7f118c004800 2024-04-24T23:49:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.278+0000 7f119d7fa640 1 -- 172.21.15.16:0/3967772770 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f119800fd20 con 0x7f118c004800 2024-04-24T23:49:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.279+0000 7f11a6d7e640 1 -- 172.21.15.16:0/3967772770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11a01538c0 con 0x7f118c004800 2024-04-24T23:49:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.280+0000 7f11a6d7e640 1 -- 172.21.15.16:0/3967772770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11a0068f90 con 0x7f118c004800 2024-04-24T23:49:46.285 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.280+0000 7f119d7fa640 1 -- 172.21.15.16:0/3967772770 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f119801d460 con 0x7f118c004800 2024-04-24T23:49:46.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.280+0000 7f119d7fa640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f118006be60 0x7f118006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.280+0000 7f119d7fa640 1 -- 172.21.15.16:0/3967772770 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f119808b1d0 con 0x7f118c004800 2024-04-24T23:49:46.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.281+0000 7f119ffff640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f118006be60 0x7f118006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.281+0000 7f119ffff640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f118006be60 0x7f118006e320 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1190035000 tx=0x7f119002f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:46.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.285+0000 7f119d7fa640 1 -- 172.21.15.16:0/3967772770 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1198059810 con 0x7f118c004800 2024-04-24T23:49:46.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.470+0000 7f11a6d7e640 1 -- 172.21.15.16:0/3967772770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f11a0153690 con 0x7f118c004800 2024-04-24T23:49:46.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.471+0000 7f119d7fa640 1 -- 172.21.15.16:0/3967772770 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f1198059110 con 0x7f118c004800 2024-04-24T23:49:46.471 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:46.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.473+0000 7f11a6d7e640 1 -- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f118006be60 msgr2=0x7f118006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.473+0000 7f11a6d7e640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f118006be60 0x7f118006e320 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1190035000 tx=0x7f119002f000 comp rx=0 tx=0).stop 2024-04-24T23:49:46.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.473+0000 7f11a6d7e640 1 -- 172.21.15.16:0/3967772770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f118c004800 msgr2=0x7f11a0141750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.473+0000 7f11a6d7e640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f118c004800 0x7f11a0141750 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1198007310 tx=0x7f11980077d0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.473+0000 7f11a6d7e640 1 -- 172.21.15.16:0/3967772770 shutdown_connections 2024-04-24T23:49:46.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.473+0000 7f11a6d7e640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f118006be60 0x7f118006e320 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.473+0000 7f11a6d7e640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f11a01f70f0 0x7f11a01f3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.473+0000 7f11a6d7e640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f11a0150ee0 0x7f11a015ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.473+0000 7f11a6d7e640 1 --2- 172.21.15.16:0/3967772770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f118c004800 0x7f11a0141750 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.473+0000 7f11a6d7e640 1 -- 172.21.15.16:0/3967772770 >> 172.21.15.16:0/3967772770 conn(0x7f11a00affc0 msgr2=0x7f11a006c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:46.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.473+0000 7f11a6d7e640 1 -- 172.21.15.16:0/3967772770 shutdown_connections 2024-04-24T23:49:46.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.473+0000 7f11a6d7e640 1 -- 172.21.15.16:0/3967772770 wait complete. 2024-04-24T23:49:46.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:46.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.588+0000 7f8aa0d34640 1 Processor -- start 2024-04-24T23:49:46.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.588+0000 7f8aa0d34640 1 -- start start 2024-04-24T23:49:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.589+0000 7f8aa0d34640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8a9c06bc40 0x7f8a9c06c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.589+0000 7f8aa0d34640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a9c0d24d0 con 0x7f8a9c15dfd0 2024-04-24T23:49:46.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.589+0000 7f8aa0d34640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a9c0d27b0 con 0x7f8a9c0b7050 2024-04-24T23:49:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.589+0000 7f8aa0d34640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8a9c06ba00 con 0x7f8a9c06bc40 2024-04-24T23:49:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.589+0000 7f8a9a59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8a9c15dfd0 0x7f8a9c06b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52388/0 (socket says 172.21.15.16:52388) 2024-04-24T23:49:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.589+0000 7f8a9a59c640 1 -- 172.21.15.16:0/2813586310 learned_addr learned my addr 172.21.15.16:0/2813586310 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a99d9b640 1 --2- 172.21.15.16:0/2813586310 >> v2:172.21.15.156:3300/0 conn(0x7f8a9c06bc40 0x7f8a9c06c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 -- 172.21.15.16:0/2813586310 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2416631225 0 0) 0x7f8a9c0d24d0 con 0x7f8a9c15dfd0 2024-04-24T23:49:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 -- 172.21.15.16:0/2813586310 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a88003520 con 0x7f8a9c15dfd0 2024-04-24T23:49:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 -- 172.21.15.16:0/2813586310 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3772404814 0 0) 0x7f8a88003520 con 0x7f8a9c15dfd0 2024-04-24T23:49:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 -- 172.21.15.16:0/2813586310 >> v2:172.21.15.156:3300/0 conn(0x7f8a9c06bc40 msgr2=0x7f8a9c06c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 --2- 172.21.15.16:0/2813586310 >> v2:172.21.15.156:3300/0 conn(0x7f8a9c06bc40 0x7f8a9c06c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 -- 172.21.15.16:0/2813586310 >> v1:172.21.15.156:6789/0 conn(0x7f8a9c0b7050 legacy=0x7f8a9c0b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 -- 172.21.15.16:0/2813586310 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a9c0d2a90 con 0x7f8a9c15dfd0 2024-04-24T23:49:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 -- 172.21.15.16:0/2813586310 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f8a94001670 con 0x7f8a9c15dfd0 2024-04-24T23:49:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 -- 172.21.15.16:0/2813586310 >> v1:172.21.15.16:6789/0 conn(0x7f8a9c15dfd0 legacy=0x7f8a9c06b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 --2- 172.21.15.16:0/2813586310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a880039b0 0x7f8a88003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 --2- 172.21.15.16:0/2813586310 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a88004800 0x7f8a88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 --2- 172.21.15.16:0/2813586310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a880051c0 0x7f8a8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 -- 172.21.15.16:0/2813586310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c06ba00 con 0x7f8a880051c0 2024-04-24T23:49:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 -- 172.21.15.16:0/2813586310 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8a8800c160 con 0x7f8a880039b0 2024-04-24T23:49:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9959a640 1 -- 172.21.15.16:0/2813586310 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8a8800c360 con 0x7f8a88004800 2024-04-24T23:49:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9ad9d640 1 --2- 172.21.15.16:0/2813586310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a880051c0 0x7f8a8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9a59c640 1 --2- 172.21.15.16:0/2813586310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a880039b0 0x7f8a88003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a99d9b640 1 --2- 172.21.15.16:0/2813586310 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a88004800 0x7f8a88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9ad9d640 1 -- 172.21.15.16:0/2813586310 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a88004800 msgr2=0x7f8a88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9ad9d640 1 --2- 172.21.15.16:0/2813586310 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a88004800 0x7f8a88004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9ad9d640 1 -- 172.21.15.16:0/2813586310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a880039b0 msgr2=0x7f8a88003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9ad9d640 1 --2- 172.21.15.16:0/2813586310 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a880039b0 0x7f8a88003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.590+0000 7f8a9ad9d640 1 -- 172.21.15.16:0/2813586310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a88003520 con 0x7f8a880051c0 2024-04-24T23:49:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.591+0000 7f8a9ad9d640 1 -- 172.21.15.16:0/2813586310 reap_dead start 2024-04-24T23:49:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.591+0000 7f8a9ad9d640 1 --2- 172.21.15.16:0/2813586310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a880051c0 0x7f8a8800b9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c001e50 tx=0x7f8a88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61f2458d49d78da server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.591+0000 7f8a9959a640 1 -- 172.21.15.16:0/2813586310 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c001a70 con 0x7f8a880051c0 2024-04-24T23:49:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.591+0000 7f8a9959a640 1 -- 172.21.15.16:0/2813586310 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a8c003970 con 0x7f8a880051c0 2024-04-24T23:49:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.591+0000 7f8a9959a640 1 -- 172.21.15.16:0/2813586310 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c003d60 con 0x7f8a880051c0 2024-04-24T23:49:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.592+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2813586310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a880051c0 msgr2=0x7f8a8800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.592+0000 7f8aa0d34640 1 --2- 172.21.15.16:0/2813586310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a880051c0 0x7f8a8800b9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c001e50 tx=0x7f8a88005830 comp rx=0 tx=0).stop 2024-04-24T23:49:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.592+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2813586310 shutdown_connections 2024-04-24T23:49:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.592+0000 7f8aa0d34640 1 --2- 172.21.15.16:0/2813586310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a880051c0 0x7f8a8800b9c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.592+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2813586310 >> 172.21.15.16:0/2813586310 conn(0x7f8a9c0d3dc0 msgr2=0x7f8a9c0d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.592+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2813586310 shutdown_connections 2024-04-24T23:49:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.592+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2813586310 wait complete. 2024-04-24T23:49:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.593+0000 7f8aa0d34640 1 Processor -- start 2024-04-24T23:49:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.593+0000 7f8aa0d34640 1 -- start start 2024-04-24T23:49:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8aa0d34640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a880051c0 0x7f8a9c0cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8aa0d34640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a9c157f50 0x7f8a9c157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8aa0d34640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a9c1f37b0 0x7f8a9c15c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8aa0d34640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c0cead0 con 0x7f8a880051c0 2024-04-24T23:49:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8aa0d34640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c0d3050 con 0x7f8a9c1f37b0 2024-04-24T23:49:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8aa0d34640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8a9c06c0d0 con 0x7f8a9c157f50 2024-04-24T23:49:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8a9a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a880051c0 0x7f8a9c0cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8a9a59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a880051c0 0x7f8a9c0cdb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60020/0 (socket says 172.21.15.16:60020) 2024-04-24T23:49:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8a9a59c640 1 -- 172.21.15.16:0/2149840383 learned_addr learned my addr 172.21.15.16:0/2149840383 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8a99d9b640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a9c157f50 0x7f8a9c157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8a9ad9d640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a9c1f37b0 0x7f8a9c15c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8a9a59c640 1 -- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a9c157f50 msgr2=0x7f8a9c157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8a9a59c640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a9c157f50 0x7f8a9c157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8a9a59c640 1 -- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a9c1f37b0 msgr2=0x7f8a9c15c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8a9a59c640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a9c1f37b0 0x7f8a9c15c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8a9a59c640 1 -- 172.21.15.16:0/2149840383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a9c15f560 con 0x7f8a880051c0 2024-04-24T23:49:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8a99d9b640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a9c157f50 0x7f8a9c157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.594+0000 7f8a9ad9d640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a9c1f37b0 0x7f8a9c15c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.595+0000 7f8a9a59c640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a880051c0 0x7f8a9c0cdb00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8a94007340 tx=0x7f8a94007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.595+0000 7f8a92ffd640 1 -- 172.21.15.16:0/2149840383 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400f910 con 0x7f8a880051c0 2024-04-24T23:49:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.595+0000 7f8a92ffd640 1 -- 172.21.15.16:0/2149840383 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a94018070 con 0x7f8a880051c0 2024-04-24T23:49:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.595+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2149840383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a9c15f840 con 0x7f8a880051c0 2024-04-24T23:49:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.595+0000 7f8a92ffd640 1 -- 172.21.15.16:0/2149840383 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400fd10 con 0x7f8a880051c0 2024-04-24T23:49:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.595+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2149840383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a9c15fe00 con 0x7f8a880051c0 2024-04-24T23:49:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.596+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2149840383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a9c068f90 con 0x7f8a880051c0 2024-04-24T23:49:46.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.596+0000 7f8a92ffd640 1 -- 172.21.15.16:0/2149840383 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8a9401d460 con 0x7f8a880051c0 2024-04-24T23:49:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.597+0000 7f8a92ffd640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a7c06be60 0x7f8a7c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.597+0000 7f8a92ffd640 1 -- 172.21.15.16:0/2149840383 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8a9408b400 con 0x7f8a880051c0 2024-04-24T23:49:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.597+0000 7f8a99d9b640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a7c06be60 0x7f8a7c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.598+0000 7f8a99d9b640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a7c06be60 0x7f8a7c06e320 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c0cd890 tx=0x7f8a84008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.601+0000 7f8a92ffd640 1 -- 172.21.15.16:0/2149840383 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8a940599b0 con 0x7f8a880051c0 2024-04-24T23:49:46.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.783+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2149840383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8a9c067730 con 0x7f8a880051c0 2024-04-24T23:49:46.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.783+0000 7f8a92ffd640 1 -- 172.21.15.16:0/2149840383 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f8a940592b0 con 0x7f8a880051c0 2024-04-24T23:49:46.783 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:46.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.785+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a7c06be60 msgr2=0x7f8a7c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.785+0000 7f8aa0d34640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a7c06be60 0x7f8a7c06e320 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c0cd890 tx=0x7f8a84008040 comp rx=0 tx=0).stop 2024-04-24T23:49:46.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.785+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2149840383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a880051c0 msgr2=0x7f8a9c0cdb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.785+0000 7f8aa0d34640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a880051c0 0x7f8a9c0cdb00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8a94007340 tx=0x7f8a94007800 comp rx=0 tx=0).stop 2024-04-24T23:49:46.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.785+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2149840383 shutdown_connections 2024-04-24T23:49:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.785+0000 7f8aa0d34640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a7c06be60 0x7f8a7c06e320 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.785+0000 7f8aa0d34640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a9c1f37b0 0x7f8a9c15c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.785+0000 7f8aa0d34640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a9c157f50 0x7f8a9c157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.785+0000 7f8aa0d34640 1 --2- 172.21.15.16:0/2149840383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a880051c0 0x7f8a9c0cdb00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.785+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2149840383 >> 172.21.15.16:0/2149840383 conn(0x7f8a9c0d3dc0 msgr2=0x7f8a9c0b88c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.785+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2149840383 shutdown_connections 2024-04-24T23:49:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.785+0000 7f8aa0d34640 1 -- 172.21.15.16:0/2149840383 wait complete. 2024-04-24T23:49:46.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.900+0000 7f6aabe9e640 1 Processor -- start 2024-04-24T23:49:46.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.900+0000 7f6aabe9e640 1 -- start start 2024-04-24T23:49:46.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.901+0000 7f6aabe9e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6aa415dfd0 0x7f6aa406b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.901+0000 7f6aabe9e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aa40cfde0 con 0x7f6aa406bc40 2024-04-24T23:49:46.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.901+0000 7f6aabe9e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6aa40d24d0 con 0x7f6aa40b8060 2024-04-24T23:49:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.901+0000 7f6aabe9e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa40d27b0 con 0x7f6aa415dfd0 2024-04-24T23:49:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.901+0000 7f6aa9439640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6aa406bc40 0x7f6aa40cefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42216/0 (socket says 172.21.15.16:42216) 2024-04-24T23:49:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.901+0000 7f6aa9439640 1 -- 172.21.15.16:0/2616825317 learned_addr learned my addr 172.21.15.16:0/2616825317 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa9c3a640 1 --2- 172.21.15.16:0/2616825317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6aa415dfd0 0x7f6aa406b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1302651671 0 0) 0x7f6aa40cfde0 con 0x7f6aa406bc40 2024-04-24T23:49:46.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a8c003520 con 0x7f6aa406bc40 2024-04-24T23:49:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2554278668 0 0) 0x7f6aa40d24d0 con 0x7f6aa40b8060 2024-04-24T23:49:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6aa40cfde0 con 0x7f6aa40b8060 2024-04-24T23:49:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 372273920 0 0) 0x7f6a8c003520 con 0x7f6aa406bc40 2024-04-24T23:49:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6aa415dfd0 msgr2=0x7f6aa406b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 --2- 172.21.15.16:0/2616825317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6aa415dfd0 0x7f6aa406b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 >> v1:172.21.15.156:6789/0 conn(0x7f6aa40b8060 legacy=0x7f6aa40cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa40d2a90 con 0x7f6aa406bc40 2024-04-24T23:49:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f6a98001670 con 0x7f6aa406bc40 2024-04-24T23:49:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa9c3a640 1 --2- 172.21.15.16:0/2616825317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6aa415dfd0 0x7f6aa406b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 >> v1:172.21.15.16:6789/0 conn(0x7f6aa406bc40 legacy=0x7f6aa40cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 --2- 172.21.15.16:0/2616825317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c003b70 0x7f6a8c003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 --2- 172.21.15.16:0/2616825317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a8c004800 0x7f6a8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 --2- 172.21.15.16:0/2616825317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6aa415dfd0 0x7f6a8c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa40d27b0 con 0x7f6a8c003b70 2024-04-24T23:49:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8c00baa0 con 0x7f6a8c004800 2024-04-24T23:49:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6a8c00bca0 con 0x7f6aa415dfd0 2024-04-24T23:49:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa9c3a640 1 --2- 172.21.15.16:0/2616825317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c003b70 0x7f6a8c003f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa9439640 1 --2- 172.21.15.16:0/2616825317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a8c004800 0x7f6a8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aaa43b640 1 --2- 172.21.15.16:0/2616825317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6aa415dfd0 0x7f6a8c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa9c3a640 1 -- 172.21.15.16:0/2616825317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6aa415dfd0 msgr2=0x7f6a8c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa9c3a640 1 --2- 172.21.15.16:0/2616825317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6aa415dfd0 0x7f6a8c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa9c3a640 1 -- 172.21.15.16:0/2616825317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a8c004800 msgr2=0x7f6a8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa9c3a640 1 --2- 172.21.15.16:0/2616825317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a8c004800 0x7f6a8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa9c3a640 1 -- 172.21.15.16:0/2616825317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a8c003520 con 0x7f6a8c003b70 2024-04-24T23:49:46.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.902+0000 7f6aa9439640 1 --2- 172.21.15.16:0/2616825317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a8c004800 0x7f6a8c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.903+0000 7f6aa9c3a640 1 --2- 172.21.15.16:0/2616825317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c003b70 0x7f6a8c003f70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400e320 tx=0x7f6a8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=773843dfc1010149 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.903+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a94033070 con 0x7f6a8c003b70 2024-04-24T23:49:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.903+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a9402f630 con 0x7f6a8c003b70 2024-04-24T23:49:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.903+0000 7f6aa8c38640 1 -- 172.21.15.16:0/2616825317 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a94009de0 con 0x7f6a8c003b70 2024-04-24T23:49:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.904+0000 7f6aabe9e640 1 -- 172.21.15.16:0/2616825317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c003b70 msgr2=0x7f6a8c003f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.904+0000 7f6aabe9e640 1 --2- 172.21.15.16:0/2616825317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c003b70 0x7f6a8c003f70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400e320 tx=0x7f6a8c005830 comp rx=0 tx=0).stop 2024-04-24T23:49:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.904+0000 7f6aaa43b640 1 -- 172.21.15.16:0/2616825317 reap_dead start 2024-04-24T23:49:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.904+0000 7f6aabe9e640 1 -- 172.21.15.16:0/2616825317 shutdown_connections 2024-04-24T23:49:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.904+0000 7f6aabe9e640 1 -- 172.21.15.16:0/2616825317 >> 172.21.15.16:0/2616825317 conn(0x7f6aa40d3dc0 msgr2=0x7f6aa40d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.904+0000 7f6aabe9e640 1 -- 172.21.15.16:0/2616825317 shutdown_connections 2024-04-24T23:49:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.904+0000 7f6aabe9e640 1 -- 172.21.15.16:0/2616825317 wait complete. 2024-04-24T23:49:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.905+0000 7f6aabe9e640 1 Processor -- start 2024-04-24T23:49:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.905+0000 7f6aabe9e640 1 -- start start 2024-04-24T23:49:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aabe9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c003b70 0x7f6aa414a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aabe9e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6aa4160db0 0x7f6aa40c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aabe9e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6aa41479b0 0x7f6aa415f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aabe9e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa40cead0 con 0x7f6a8c003b70 2024-04-24T23:49:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aabe9e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6aa40d3050 con 0x7f6aa4160db0 2024-04-24T23:49:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aabe9e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6aa406c0a0 con 0x7f6aa41479b0 2024-04-24T23:49:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aa9c3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c003b70 0x7f6aa414a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aa9c3a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c003b70 0x7f6aa414a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40534/0 (socket says 172.21.15.16:40534) 2024-04-24T23:49:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aa9c3a640 1 -- 172.21.15.16:0/432568604 learned_addr learned my addr 172.21.15.16:0/432568604 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aa9c3a640 1 -- 172.21.15.16:0/432568604 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6aa41479b0 msgr2=0x7f6aa415f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aa9c3a640 1 --2- 172.21.15.16:0/432568604 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6aa41479b0 0x7f6aa415f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aa9c3a640 1 -- 172.21.15.16:0/432568604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6aa4160db0 msgr2=0x7f6aa40c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aa9439640 1 --2- 172.21.15.16:0/432568604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6aa4160db0 0x7f6aa40c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aa9c3a640 1 --2- 172.21.15.16:0/432568604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6aa4160db0 0x7f6aa40c9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aa9c3a640 1 -- 172.21.15.16:0/432568604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6aa415e600 con 0x7f6a8c003b70 2024-04-24T23:49:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aa9439640 1 --2- 172.21.15.16:0/432568604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6aa4160db0 0x7f6aa40c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.906+0000 7f6aa9c3a640 1 --2- 172.21.15.16:0/432568604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c003b70 0x7f6aa414a450 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400e440 tx=0x7f6a94009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.907+0000 7f6aa27fc640 1 -- 172.21.15.16:0/432568604 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a94033040 con 0x7f6a8c003b70 2024-04-24T23:49:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.907+0000 7f6aa27fc640 1 -- 172.21.15.16:0/432568604 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a94009a50 con 0x7f6a8c003b70 2024-04-24T23:49:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.907+0000 7f6aabe9e640 1 -- 172.21.15.16:0/432568604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6aa415e8e0 con 0x7f6a8c003b70 2024-04-24T23:49:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.907+0000 7f6aa27fc640 1 -- 172.21.15.16:0/432568604 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a94009de0 con 0x7f6a8c003b70 2024-04-24T23:49:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.907+0000 7f6aabe9e640 1 -- 172.21.15.16:0/432568604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6aa415eea0 con 0x7f6a8c003b70 2024-04-24T23:49:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.908+0000 7f6aabe9e640 1 -- 172.21.15.16:0/432568604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6aa4068f90 con 0x7f6a8c003b70 2024-04-24T23:49:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.908+0000 7f6aa27fc640 1 -- 172.21.15.16:0/432568604 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6a94036070 con 0x7f6a8c003b70 2024-04-24T23:49:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.908+0000 7f6aa27fc640 1 --2- 172.21.15.16:0/432568604 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6a9006be30 0x7f6a9006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.909+0000 7f6aa27fc640 1 -- 172.21.15.16:0/432568604 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f6a940ac040 con 0x7f6a8c003b70 2024-04-24T23:49:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.909+0000 7f6aa9439640 1 --2- 172.21.15.16:0/432568604 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6a9006be30 0x7f6a9006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.909+0000 7f6aa9439640 1 --2- 172.21.15.16:0/432568604 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6a9006be30 0x7f6a9006e2f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6a98000fd0 tx=0x7f6a98001370 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:46.913+0000 7f6aa27fc640 1 -- 172.21.15.16:0/432568604 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6a9407a680 con 0x7f6a8c003b70 2024-04-24T23:49:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.097+0000 7f6aabe9e640 1 -- 172.21.15.16:0/432568604 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f6aa4159130 con 0x7f6a8c003b70 2024-04-24T23:49:47.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.098+0000 7f6aa27fc640 1 -- 172.21.15.16:0/432568604 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f6a94079f80 con 0x7f6a8c003b70 2024-04-24T23:49:47.098 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.100+0000 7f6aabe9e640 1 -- 172.21.15.16:0/432568604 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6a9006be30 msgr2=0x7f6a9006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.100+0000 7f6aabe9e640 1 --2- 172.21.15.16:0/432568604 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6a9006be30 0x7f6a9006e2f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6a98000fd0 tx=0x7f6a98001370 comp rx=0 tx=0).stop 2024-04-24T23:49:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.100+0000 7f6aabe9e640 1 -- 172.21.15.16:0/432568604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c003b70 msgr2=0x7f6aa414a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.100+0000 7f6aabe9e640 1 --2- 172.21.15.16:0/432568604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c003b70 0x7f6aa414a450 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400e440 tx=0x7f6a94009750 comp rx=0 tx=0).stop 2024-04-24T23:49:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.100+0000 7f6aabe9e640 1 -- 172.21.15.16:0/432568604 shutdown_connections 2024-04-24T23:49:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.100+0000 7f6aabe9e640 1 --2- 172.21.15.16:0/432568604 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6a9006be30 0x7f6a9006e2f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.100+0000 7f6aabe9e640 1 --2- 172.21.15.16:0/432568604 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6aa41479b0 0x7f6aa415f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.100+0000 7f6aabe9e640 1 --2- 172.21.15.16:0/432568604 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6aa4160db0 0x7f6aa40c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.100+0000 7f6aabe9e640 1 --2- 172.21.15.16:0/432568604 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a8c003b70 0x7f6aa414a450 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.100+0000 7f6aabe9e640 1 -- 172.21.15.16:0/432568604 >> 172.21.15.16:0/432568604 conn(0x7f6aa40d3dc0 msgr2=0x7f6aa40b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:47.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.100+0000 7f6aabe9e640 1 -- 172.21.15.16:0/432568604 shutdown_connections 2024-04-24T23:49:47.101 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.100+0000 7f6aabe9e640 1 -- 172.21.15.16:0/432568604 wait complete. 2024-04-24T23:49:47.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_recovery_fs.TestFSRecovery.test_recover_fs_after_fsmap_removal' 2024-04-24T23:49:47.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.212+0000 7f5729f42640 1 Processor -- start 2024-04-24T23:49:47.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.212+0000 7f5729f42640 1 -- start start 2024-04-24T23:49:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.213+0000 7f5729f42640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f57240b2bb0 0x7f57240cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.213+0000 7f5729f42640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57240b3390 con 0x7f572415e020 2024-04-24T23:49:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.213+0000 7f5729f42640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f572406bfc0 con 0x7f57240d1fa0 2024-04-24T23:49:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.213+0000 7f5729f42640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f572406c2a0 con 0x7f57240b2bb0 2024-04-24T23:49:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.213+0000 7f57237fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f572415e020 0x7f57240cf270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42218/0 (socket says 172.21.15.16:42218) 2024-04-24T23:49:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.213+0000 7f57237fe640 1 -- 172.21.15.16:0/1288478154 learned_addr learned my addr 172.21.15.16:0/1288478154 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.214+0000 7f5722ffd640 1 --2- 172.21.15.16:0/1288478154 >> v2:172.21.15.156:3300/0 conn(0x7f57240b2bb0 0x7f57240cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.214+0000 7f57227fc640 1 -- 172.21.15.16:0/1288478154 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3487579608 0 0) 0x7f57240b3390 con 0x7f572415e020 2024-04-24T23:49:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.214+0000 7f57227fc640 1 -- 172.21.15.16:0/1288478154 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f570c003520 con 0x7f572415e020 2024-04-24T23:49:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.214+0000 7f5722ffd640 1 -- 172.21.15.16:0/1288478154 >> v1:172.21.15.156:6789/0 conn(0x7f57240d1fa0 legacy=0x7f57240d23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.214+0000 7f5722ffd640 1 -- 172.21.15.16:0/1288478154 >> v1:172.21.15.16:6789/0 conn(0x7f572415e020 legacy=0x7f57240cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.214+0000 7f57227fc640 1 -- 172.21.15.16:0/1288478154 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2220126212 0 0) 0x7f570c003520 con 0x7f572415e020 2024-04-24T23:49:47.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.214+0000 7f5722ffd640 1 -- 172.21.15.16:0/1288478154 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f572406c580 con 0x7f57240b2bb0 2024-04-24T23:49:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.215+0000 7f5722ffd640 1 --2- 172.21.15.16:0/1288478154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57240b2bb0 0x7f57240cf980 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f57180097a0 tx=0x7f571802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=272d9af3a97a105f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.215+0000 7f57227fc640 1 -- 172.21.15.16:0/1288478154 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5718037070 con 0x7f57240b2bb0 2024-04-24T23:49:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.215+0000 7f57227fc640 1 -- 172.21.15.16:0/1288478154 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5718032440 con 0x7f57240b2bb0 2024-04-24T23:49:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.215+0000 7f57227fc640 1 -- 172.21.15.16:0/1288478154 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5718032810 con 0x7f57240b2bb0 2024-04-24T23:49:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.215+0000 7f5729f42640 1 -- 172.21.15.16:0/1288478154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57240b2bb0 msgr2=0x7f57240cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.215+0000 7f5729f42640 1 --2- 172.21.15.16:0/1288478154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57240b2bb0 0x7f57240cf980 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f57180097a0 tx=0x7f571802a1a0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.215+0000 7f5729f42640 1 -- 172.21.15.16:0/1288478154 shutdown_connections 2024-04-24T23:49:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.215+0000 7f5729f42640 1 --2- 172.21.15.16:0/1288478154 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57240b2bb0 0x7f57240cf980 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.215+0000 7f5729f42640 1 -- 172.21.15.16:0/1288478154 >> 172.21.15.16:0/1288478154 conn(0x7f57240d3e10 msgr2=0x7f57240d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.215+0000 7f5729f42640 1 -- 172.21.15.16:0/1288478154 shutdown_connections 2024-04-24T23:49:47.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.215+0000 7f5729f42640 1 -- 172.21.15.16:0/1288478154 wait complete. 2024-04-24T23:49:47.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.216+0000 7f5729f42640 1 Processor -- start 2024-04-24T23:49:47.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.217+0000 7f5729f42640 1 -- start start 2024-04-24T23:49:47.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.217+0000 7f5729f42640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57240b2bb0 0x7f57240cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:47.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.217+0000 7f5729f42640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57240d1fa0 0x7f57241f4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:47.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.217+0000 7f5729f42640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f572415e020 0x7f5724160af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.217+0000 7f5729f42640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f572415e400 con 0x7f572415e020 2024-04-24T23:49:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.217+0000 7f5729f42640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f57240d1a80 con 0x7f57240b2bb0 2024-04-24T23:49:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.217+0000 7f5729f42640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f57240cfec0 con 0x7f57240d1fa0 2024-04-24T23:49:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f5723fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f572415e020 0x7f5724160af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f5723fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f572415e020 0x7f5724160af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40548/0 (socket says 172.21.15.16:40548) 2024-04-24T23:49:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f5723fff640 1 -- 172.21.15.16:0/3065781317 learned_addr learned my addr 172.21.15.16:0/3065781317 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f57237fe640 1 --2- 172.21.15.16:0/3065781317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57240b2bb0 0x7f57240cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f5722ffd640 1 --2- 172.21.15.16:0/3065781317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57240d1fa0 0x7f57241f4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f5723fff640 1 -- 172.21.15.16:0/3065781317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57240d1fa0 msgr2=0x7f57241f4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f5723fff640 1 --2- 172.21.15.16:0/3065781317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57240d1fa0 0x7f57241f4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f5723fff640 1 -- 172.21.15.16:0/3065781317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57240b2bb0 msgr2=0x7f57240cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f5723fff640 1 --2- 172.21.15.16:0/3065781317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57240b2bb0 0x7f57240cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f5723fff640 1 -- 172.21.15.16:0/3065781317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57240b4400 con 0x7f572415e020 2024-04-24T23:49:47.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f5723fff640 1 --2- 172.21.15.16:0/3065781317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f572415e020 0x7f5724160af0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f571c00c7a0 tx=0x7f571c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:47.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f570bfff640 1 -- 172.21.15.16:0/3065781317 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c003b70 con 0x7f572415e020 2024-04-24T23:49:47.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f570bfff640 1 -- 172.21.15.16:0/3065781317 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f571c003d30 con 0x7f572415e020 2024-04-24T23:49:47.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f5729f42640 1 -- 172.21.15.16:0/3065781317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57241f52c0 con 0x7f572415e020 2024-04-24T23:49:47.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f570bfff640 1 -- 172.21.15.16:0/3065781317 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f571c00aa90 con 0x7f572415e020 2024-04-24T23:49:47.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.218+0000 7f5729f42640 1 -- 172.21.15.16:0/3065781317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57240cb2d0 con 0x7f572415e020 2024-04-24T23:49:47.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.219+0000 7f5729f42640 1 -- 172.21.15.16:0/3065781317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f572406dfe0 con 0x7f572415e020 2024-04-24T23:49:47.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.220+0000 7f570bfff640 1 -- 172.21.15.16:0/3065781317 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f571c00ac30 con 0x7f572415e020 2024-04-24T23:49:47.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.220+0000 7f570bfff640 1 --2- 172.21.15.16:0/3065781317 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f571006be60 0x7f571006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:47.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.220+0000 7f570bfff640 1 -- 172.21.15.16:0/3065781317 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f571c08b030 con 0x7f572415e020 2024-04-24T23:49:47.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.220+0000 7f57237fe640 1 --2- 172.21.15.16:0/3065781317 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f571006be60 0x7f571006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:47.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.221+0000 7f57237fe640 1 --2- 172.21.15.16:0/3065781317 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f571006be60 0x7f571006e320 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5714005880 tx=0x7f5714009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:47.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.224+0000 7f570bfff640 1 -- 172.21.15.16:0/3065781317 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f57241f52c0 con 0x7f572415e020 2024-04-24T23:49:47.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.476+0000 7f5729f42640 1 -- 172.21.15.16:0/3065781317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_recovery_fs.TestFSRecovery.test_recover_fs_after_fsmap_removal"]} v 0) -- 0x7f57240b6f90 con 0x7f572415e020 2024-04-24T23:49:47.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.647+0000 7f570bfff640 1 -- 172.21.15.16:0/3065781317 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_recovery_fs.TestFSRecovery.test_recover_fs_after_fsmap_removal"]}]=0 v72) ==== 162+0+0 (secure 0 0 0) 0x7f571c058f90 con 0x7f572415e020 2024-04-24T23:49:47.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.650+0000 7f5729f42640 1 -- 172.21.15.16:0/3065781317 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f571006be60 msgr2=0x7f571006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.650+0000 7f5729f42640 1 --2- 172.21.15.16:0/3065781317 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f571006be60 0x7f571006e320 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5714005880 tx=0x7f5714009040 comp rx=0 tx=0).stop 2024-04-24T23:49:47.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.650+0000 7f5729f42640 1 -- 172.21.15.16:0/3065781317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f572415e020 msgr2=0x7f5724160af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.650+0000 7f5729f42640 1 --2- 172.21.15.16:0/3065781317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f572415e020 0x7f5724160af0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f571c00c7a0 tx=0x7f571c00cc60 comp rx=0 tx=0).stop 2024-04-24T23:49:47.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.650+0000 7f5729f42640 1 -- 172.21.15.16:0/3065781317 shutdown_connections 2024-04-24T23:49:47.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.650+0000 7f5729f42640 1 --2- 172.21.15.16:0/3065781317 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f571006be60 0x7f571006e320 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.650+0000 7f5729f42640 1 --2- 172.21.15.16:0/3065781317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f572415e020 0x7f5724160af0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.650+0000 7f5729f42640 1 --2- 172.21.15.16:0/3065781317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f57240d1fa0 0x7f57241f4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.650+0000 7f5729f42640 1 --2- 172.21.15.16:0/3065781317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f57240b2bb0 0x7f57240cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.650+0000 7f5729f42640 1 -- 172.21.15.16:0/3065781317 >> 172.21.15.16:0/3065781317 conn(0x7f57240d3e10 msgr2=0x7f572415b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:47.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.650+0000 7f5729f42640 1 -- 172.21.15.16:0/3065781317 shutdown_connections 2024-04-24T23:49:47.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.650+0000 7f5729f42640 1 -- 172.21.15.16:0/3065781317 wait complete. 2024-04-24T23:49:47.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-04-24T23:49:47.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.764+0000 7fe269a5d640 1 Processor -- start 2024-04-24T23:49:47.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.765+0000 7fe269a5d640 1 -- start start 2024-04-24T23:49:47.766 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.765+0000 7fe269a5d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2640d12d0 0x7fe2640d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:47.767 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.765+0000 7fe269a5d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe26415e5c0 0x7fe2640d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.765+0000 7fe269a5d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe26415ec30 con 0x7fe2640cd790 2024-04-24T23:49:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.765+0000 7fe269a5d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fe26415f4d0 con 0x7fe26415e5c0 2024-04-24T23:49:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.765+0000 7fe269a5d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe26415ef10 con 0x7fe2640d12d0 2024-04-24T23:49:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.766+0000 7fe2627fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe2640cd790 0x7fe2640c8580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42226/0 (socket says 172.21.15.16:42226) 2024-04-24T23:49:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.766+0000 7fe2627fc640 1 -- 172.21.15.16:0/4204156050 learned_addr learned my addr 172.21.15.16:0/4204156050 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.766+0000 7fe2637fe640 1 --2- 172.21.15.16:0/4204156050 >> v2:172.21.15.156:3300/0 conn(0x7fe26415e5c0 0x7fe2640d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.766+0000 7fe262ffd640 1 --2- 172.21.15.16:0/4204156050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2640d12d0 0x7fe2640d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.766+0000 7fe261ffb640 1 -- 172.21.15.16:0/4204156050 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 90563346 0 0) 0x7fe26415ec30 con 0x7fe2640cd790 2024-04-24T23:49:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.766+0000 7fe261ffb640 1 -- 172.21.15.16:0/4204156050 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe24c003520 con 0x7fe2640cd790 2024-04-24T23:49:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.766+0000 7fe262ffd640 1 -- 172.21.15.16:0/4204156050 >> v2:172.21.15.156:3300/0 conn(0x7fe26415e5c0 msgr2=0x7fe2640d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.766+0000 7fe262ffd640 1 --2- 172.21.15.16:0/4204156050 >> v2:172.21.15.156:3300/0 conn(0x7fe26415e5c0 0x7fe2640d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.766+0000 7fe262ffd640 1 -- 172.21.15.16:0/4204156050 >> v1:172.21.15.16:6789/0 conn(0x7fe2640cd790 legacy=0x7fe2640c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.766+0000 7fe261ffb640 1 -- 172.21.15.16:0/4204156050 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3514228066 0 0) 0x7fe24c003520 con 0x7fe2640cd790 2024-04-24T23:49:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.766+0000 7fe262ffd640 1 -- 172.21.15.16:0/4204156050 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26415f1f0 con 0x7fe2640d12d0 2024-04-24T23:49:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.766+0000 7fe2637fe640 1 --2- 172.21.15.16:0/4204156050 >> v2:172.21.15.156:3300/0 conn(0x7fe26415e5c0 0x7fe2640d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.767+0000 7fe262ffd640 1 --2- 172.21.15.16:0/4204156050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2640d12d0 0x7fe2640d16b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe2540097a0 tx=0x7fe25402a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ab7c2f5c1e29a2b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.767+0000 7fe261ffb640 1 -- 172.21.15.16:0/4204156050 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe254037070 con 0x7fe2640d12d0 2024-04-24T23:49:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.767+0000 7fe261ffb640 1 -- 172.21.15.16:0/4204156050 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe25402aa90 con 0x7fe2640d12d0 2024-04-24T23:49:47.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.767+0000 7fe261ffb640 1 -- 172.21.15.16:0/4204156050 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe258001670 con 0x7fe2640d12d0 2024-04-24T23:49:47.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.768+0000 7fe269a5d640 1 -- 172.21.15.16:0/4204156050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2640d12d0 msgr2=0x7fe2640d16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.768+0000 7fe269a5d640 1 --2- 172.21.15.16:0/4204156050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2640d12d0 0x7fe2640d16b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fe2540097a0 tx=0x7fe25402a000 comp rx=0 tx=0).stop 2024-04-24T23:49:47.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.768+0000 7fe269a5d640 1 -- 172.21.15.16:0/4204156050 shutdown_connections 2024-04-24T23:49:47.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.768+0000 7fe269a5d640 1 --2- 172.21.15.16:0/4204156050 >> v2:172.21.15.156:3300/0 conn(0x7fe26415e5c0 0x7fe2640d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.768+0000 7fe269a5d640 1 --2- 172.21.15.16:0/4204156050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2640d12d0 0x7fe2640d16b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.768+0000 7fe269a5d640 1 -- 172.21.15.16:0/4204156050 >> 172.21.15.16:0/4204156050 conn(0x7fe264072e70 msgr2=0x7fe264073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:47.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.768+0000 7fe269a5d640 1 -- 172.21.15.16:0/4204156050 shutdown_connections 2024-04-24T23:49:47.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.768+0000 7fe269a5d640 1 -- 172.21.15.16:0/4204156050 wait complete. 2024-04-24T23:49:47.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.769+0000 7fe269a5d640 1 Processor -- start 2024-04-24T23:49:47.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.769+0000 7fe269a5d640 1 -- start start 2024-04-24T23:49:47.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe269a5d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2640cd790 0x7fe264144600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:47.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe269a5d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2640d12d0 0x7fe264157470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:47.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe269a5d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26415e5c0 0x7fe2641579b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:47.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe269a5d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe2640d27f0 con 0x7fe26415e5c0 2024-04-24T23:49:47.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe269a5d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe2640d2670 con 0x7fe2640cd790 2024-04-24T23:49:47.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe269a5d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe2640d2970 con 0x7fe2640d12d0 2024-04-24T23:49:47.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe2637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26415e5c0 0x7fe2641579b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:47.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe2637fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26415e5c0 0x7fe2641579b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40556/0 (socket says 172.21.15.16:40556) 2024-04-24T23:49:47.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe2637fe640 1 -- 172.21.15.16:0/2179590084 learned_addr learned my addr 172.21.15.16:0/2179590084 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:47.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe262ffd640 1 --2- 172.21.15.16:0/2179590084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2640cd790 0x7fe264144600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:47.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe2627fc640 1 --2- 172.21.15.16:0/2179590084 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2640d12d0 0x7fe264157470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:47.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe2637fe640 1 -- 172.21.15.16:0/2179590084 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2640d12d0 msgr2=0x7fe264157470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe2637fe640 1 --2- 172.21.15.16:0/2179590084 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2640d12d0 0x7fe264157470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe2637fe640 1 -- 172.21.15.16:0/2179590084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2640cd790 msgr2=0x7fe264144600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe2637fe640 1 --2- 172.21.15.16:0/2179590084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2640cd790 0x7fe264144600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe2637fe640 1 -- 172.21.15.16:0/2179590084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe264151d70 con 0x7fe26415e5c0 2024-04-24T23:49:47.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe2637fe640 1 --2- 172.21.15.16:0/2179590084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26415e5c0 0x7fe2641579b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00efc0 tx=0x7fe25c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:47.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.770+0000 7fe24bfff640 1 -- 172.21.15.16:0/2179590084 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c018030 con 0x7fe26415e5c0 2024-04-24T23:49:47.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.771+0000 7fe24bfff640 1 -- 172.21.15.16:0/2179590084 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe25c00cdf0 con 0x7fe26415e5c0 2024-04-24T23:49:47.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.771+0000 7fe269a5d640 1 -- 172.21.15.16:0/2179590084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe264158010 con 0x7fe26415e5c0 2024-04-24T23:49:47.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.771+0000 7fe24bfff640 1 -- 172.21.15.16:0/2179590084 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe25c003bb0 con 0x7fe26415e5c0 2024-04-24T23:49:47.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.771+0000 7fe269a5d640 1 -- 172.21.15.16:0/2179590084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2640adaf0 con 0x7fe26415e5c0 2024-04-24T23:49:47.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.772+0000 7fe269a5d640 1 -- 172.21.15.16:0/2179590084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2640ad8e0 con 0x7fe26415e5c0 2024-04-24T23:49:47.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.772+0000 7fe24bfff640 1 -- 172.21.15.16:0/2179590084 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe25c005d10 con 0x7fe26415e5c0 2024-04-24T23:49:47.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.772+0000 7fe24bfff640 1 --2- 172.21.15.16:0/2179590084 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe25006be30 0x7fe25006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:47.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.773+0000 7fe24bfff640 1 -- 172.21.15.16:0/2179590084 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe25c08bb60 con 0x7fe26415e5c0 2024-04-24T23:49:47.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.773+0000 7fe262ffd640 1 --2- 172.21.15.16:0/2179590084 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe25006be30 0x7fe25006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:47.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.773+0000 7fe262ffd640 1 --2- 172.21.15.16:0/2179590084 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe25006be30 0x7fe25006e2f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe254034000 tx=0x7fe25402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:47.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.777+0000 7fe24bfff640 1 -- 172.21.15.16:0/2179590084 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe25c022070 con 0x7fe26415e5c0 2024-04-24T23:49:47.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.961+0000 7fe269a5d640 1 -- 172.21.15.16:0/2179590084 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7fe264068f90 con 0x7fe26415e5c0 2024-04-24T23:49:47.963 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.964+0000 7fe24bfff640 1 -- 172.21.15.16:0/2179590084 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7fe25c010040 con 0x7fe26415e5c0 2024-04-24T23:49:47.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.966+0000 7fe269a5d640 1 -- 172.21.15.16:0/2179590084 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe25006be30 msgr2=0x7fe25006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.966+0000 7fe269a5d640 1 --2- 172.21.15.16:0/2179590084 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe25006be30 0x7fe25006e2f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe254034000 tx=0x7fe25402e000 comp rx=0 tx=0).stop 2024-04-24T23:49:47.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.966+0000 7fe269a5d640 1 -- 172.21.15.16:0/2179590084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26415e5c0 msgr2=0x7fe2641579b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:47.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.966+0000 7fe269a5d640 1 --2- 172.21.15.16:0/2179590084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26415e5c0 0x7fe2641579b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00efc0 tx=0x7fe25c00c490 comp rx=0 tx=0).stop 2024-04-24T23:49:47.966 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.966+0000 7fe269a5d640 1 -- 172.21.15.16:0/2179590084 shutdown_connections 2024-04-24T23:49:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.966+0000 7fe269a5d640 1 --2- 172.21.15.16:0/2179590084 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe25006be30 0x7fe25006e2f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.966+0000 7fe269a5d640 1 --2- 172.21.15.16:0/2179590084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe26415e5c0 0x7fe2641579b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.966+0000 7fe269a5d640 1 --2- 172.21.15.16:0/2179590084 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe2640d12d0 0x7fe264157470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.966+0000 7fe269a5d640 1 --2- 172.21.15.16:0/2179590084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe2640cd790 0x7fe264144600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.966+0000 7fe269a5d640 1 -- 172.21.15.16:0/2179590084 >> 172.21.15.16:0/2179590084 conn(0x7fe264072e70 msgr2=0x7fe2640b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.966+0000 7fe269a5d640 1 -- 172.21.15.16:0/2179590084 shutdown_connections 2024-04-24T23:49:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:47.966+0000 7fe269a5d640 1 -- 172.21.15.16:0/2179590084 wait complete. 2024-04-24T23:49:47.977 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T23:49:47.977 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T23:49:48.027 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T23:49:48.080 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T23:49:48.080 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:49:48.080 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:49:48.096 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:49:48.096 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:49:48.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:48.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.292+0000 7efdaca7d640 1 Processor -- start 2024-04-24T23:49:48.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.292+0000 7efdaca7d640 1 -- start start 2024-04-24T23:49:48.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.293+0000 7efdaca7d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efda815a450 0x7efda80b2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.293+0000 7efdaca7d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efda806e910 0x7efda8072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.293+0000 7efdaca7d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efda80b3300 con 0x7efda80727d0 2024-04-24T23:49:48.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.293+0000 7efdaca7d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7efda8072c20 con 0x7efda815a450 2024-04-24T23:49:48.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.293+0000 7efdaca7d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7efda80b35e0 con 0x7efda806e910 2024-04-24T23:49:48.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.293+0000 7efda659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efda815a450 0x7efda80b2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.293+0000 7efda6d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efda80727d0 0x7efda815deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42234/0 (socket says 172.21.15.16:42234) 2024-04-24T23:49:48.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.293+0000 7efda6d9d640 1 -- 172.21.15.16:0/2415898537 learned_addr learned my addr 172.21.15.16:0/2415898537 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:48.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.294+0000 7efda5d9b640 1 --2- 172.21.15.16:0/2415898537 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efda806e910 0x7efda8072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.294+0000 7efda659c640 1 -- 172.21.15.16:0/2415898537 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efda806e910 msgr2=0x7efda8072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:48.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.294+0000 7efda659c640 1 --2- 172.21.15.16:0/2415898537 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efda806e910 0x7efda8072290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.294+0000 7efda659c640 1 -- 172.21.15.16:0/2415898537 >> v1:172.21.15.16:6789/0 conn(0x7efda80727d0 legacy=0x7efda815deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.294+0000 7efda659c640 1 -- 172.21.15.16:0/2415898537 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda80b38c0 con 0x7efda815a450 2024-04-24T23:49:48.295 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.294+0000 7efda659c640 1 --2- 172.21.15.16:0/2415898537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda815a450 0x7efda80b2a20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7efd980097a0 tx=0x7efd9802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8629404de3eeb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:48.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.295+0000 7efda559a640 1 -- 172.21.15.16:0/2415898537 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98037070 con 0x7efda815a450 2024-04-24T23:49:48.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.295+0000 7efda559a640 1 -- 172.21.15.16:0/2415898537 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd9802ae50 con 0x7efda815a450 2024-04-24T23:49:48.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.295+0000 7efda559a640 1 -- 172.21.15.16:0/2415898537 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98032660 con 0x7efda815a450 2024-04-24T23:49:48.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.295+0000 7efdaca7d640 1 -- 172.21.15.16:0/2415898537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda815a450 msgr2=0x7efda80b2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.295+0000 7efdaca7d640 1 --2- 172.21.15.16:0/2415898537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda815a450 0x7efda80b2a20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7efd980097a0 tx=0x7efd9802a100 comp rx=0 tx=0).stop 2024-04-24T23:49:48.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.295+0000 7efdaca7d640 1 -- 172.21.15.16:0/2415898537 shutdown_connections 2024-04-24T23:49:48.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.295+0000 7efdaca7d640 1 --2- 172.21.15.16:0/2415898537 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efda806e910 0x7efda8072290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.295+0000 7efdaca7d640 1 --2- 172.21.15.16:0/2415898537 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda815a450 0x7efda80b2a20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.295+0000 7efdaca7d640 1 -- 172.21.15.16:0/2415898537 >> 172.21.15.16:0/2415898537 conn(0x7efda80b2000 msgr2=0x7efda80cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:48.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.295+0000 7efdaca7d640 1 -- 172.21.15.16:0/2415898537 shutdown_connections 2024-04-24T23:49:48.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.295+0000 7efdaca7d640 1 -- 172.21.15.16:0/2415898537 wait complete. 2024-04-24T23:49:48.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.296+0000 7efdaca7d640 1 Processor -- start 2024-04-24T23:49:48.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.297+0000 7efdaca7d640 1 -- start start 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.297+0000 7efdaca7d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efda806e910 0x7efda81f80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.297+0000 7efdaca7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda80727d0 0x7efda815a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.297+0000 7efdaca7d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efda815a450 0x7efda80adff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.297+0000 7efdaca7d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efda80b2f60 con 0x7efda80727d0 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.297+0000 7efdaca7d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efda806bb60 con 0x7efda806e910 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.297+0000 7efdaca7d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7efda80b30e0 con 0x7efda815a450 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.297+0000 7efda5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda80727d0 0x7efda815a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.297+0000 7efda5d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda80727d0 0x7efda815a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40580/0 (socket says 172.21.15.16:40580) 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.297+0000 7efda5d9b640 1 -- 172.21.15.16:0/1850958886 learned_addr learned my addr 172.21.15.16:0/1850958886 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.297+0000 7efda659c640 1 --2- 172.21.15.16:0/1850958886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efda806e910 0x7efda81f80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.297+0000 7efda6d9d640 1 --2- 172.21.15.16:0/1850958886 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efda815a450 0x7efda80adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.298+0000 7efda5d9b640 1 -- 172.21.15.16:0/1850958886 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efda815a450 msgr2=0x7efda80adff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.298+0000 7efda5d9b640 1 --2- 172.21.15.16:0/1850958886 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efda815a450 0x7efda80adff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.298+0000 7efda5d9b640 1 -- 172.21.15.16:0/1850958886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efda806e910 msgr2=0x7efda81f80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.298+0000 7efda5d9b640 1 --2- 172.21.15.16:0/1850958886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efda806e910 0x7efda81f80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.298+0000 7efda5d9b640 1 -- 172.21.15.16:0/1850958886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda81f8620 con 0x7efda80727d0 2024-04-24T23:49:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.298+0000 7efda5d9b640 1 --2- 172.21.15.16:0/1850958886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda80727d0 0x7efda815a830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7efd9c00d7d0 tx=0x7efd9c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.298+0000 7efd96ffd640 1 -- 172.21.15.16:0/1850958886 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c00b6c0 con 0x7efda80727d0 2024-04-24T23:49:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.298+0000 7efd96ffd640 1 -- 172.21.15.16:0/1850958886 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd9c019070 con 0x7efda80727d0 2024-04-24T23:49:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.298+0000 7efdaca7d640 1 -- 172.21.15.16:0/1850958886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efda815b130 con 0x7efda80727d0 2024-04-24T23:49:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.298+0000 7efd96ffd640 1 -- 172.21.15.16:0/1850958886 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c00bac0 con 0x7efda80727d0 2024-04-24T23:49:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.298+0000 7efdaca7d640 1 -- 172.21.15.16:0/1850958886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efda81f78e0 con 0x7efda80727d0 2024-04-24T23:49:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.299+0000 7efdaca7d640 1 -- 172.21.15.16:0/1850958886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efda815b650 con 0x7efda80727d0 2024-04-24T23:49:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.300+0000 7efd96ffd640 1 -- 172.21.15.16:0/1850958886 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7efd9c00bc60 con 0x7efda80727d0 2024-04-24T23:49:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.300+0000 7efd96ffd640 1 --2- 172.21.15.16:0/1850958886 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efd8806be30 0x7efd8806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.300+0000 7efd96ffd640 1 -- 172.21.15.16:0/1850958886 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7efd9c08b500 con 0x7efda80727d0 2024-04-24T23:49:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.300+0000 7efda659c640 1 --2- 172.21.15.16:0/1850958886 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efd8806be30 0x7efd8806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.301+0000 7efda659c640 1 --2- 172.21.15.16:0/1850958886 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efd8806be30 0x7efd8806e2f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efd98034000 tx=0x7efd9802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:48.304 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.304+0000 7efd96ffd640 1 -- 172.21.15.16:0/1850958886 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7efd9c0594f0 con 0x7efda80727d0 2024-04-24T23:49:48.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.489+0000 7efdaca7d640 1 -- 172.21.15.16:0/1850958886 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efda80731f0 con 0x7efda80727d0 2024-04-24T23:49:48.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.490+0000 7efd96ffd640 1 -- 172.21.15.16:0/1850958886 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7efd9c05d7f0 con 0x7efda80727d0 2024-04-24T23:49:48.490 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:48.490 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.492+0000 7efdaca7d640 1 -- 172.21.15.16:0/1850958886 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efd8806be30 msgr2=0x7efd8806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.492+0000 7efdaca7d640 1 --2- 172.21.15.16:0/1850958886 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efd8806be30 0x7efd8806e2f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efd98034000 tx=0x7efd9802e000 comp rx=0 tx=0).stop 2024-04-24T23:49:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.492+0000 7efdaca7d640 1 -- 172.21.15.16:0/1850958886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda80727d0 msgr2=0x7efda815a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.492+0000 7efdaca7d640 1 --2- 172.21.15.16:0/1850958886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda80727d0 0x7efda815a830 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7efd9c00d7d0 tx=0x7efd9c00dc90 comp rx=0 tx=0).stop 2024-04-24T23:49:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.492+0000 7efdaca7d640 1 -- 172.21.15.16:0/1850958886 shutdown_connections 2024-04-24T23:49:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.492+0000 7efdaca7d640 1 --2- 172.21.15.16:0/1850958886 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efd8806be30 0x7efd8806e2f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.492+0000 7efdaca7d640 1 --2- 172.21.15.16:0/1850958886 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efda815a450 0x7efda80adff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.492+0000 7efdaca7d640 1 --2- 172.21.15.16:0/1850958886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efda80727d0 0x7efda815a830 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.492+0000 7efdaca7d640 1 --2- 172.21.15.16:0/1850958886 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efda806e910 0x7efda81f80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.492+0000 7efdaca7d640 1 -- 172.21.15.16:0/1850958886 >> 172.21.15.16:0/1850958886 conn(0x7efda80b2000 msgr2=0x7efda814f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.492+0000 7efdaca7d640 1 -- 172.21.15.16:0/1850958886 shutdown_connections 2024-04-24T23:49:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.492+0000 7efdaca7d640 1 -- 172.21.15.16:0/1850958886 wait complete. 2024-04-24T23:49:48.503 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:48.503 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.606+0000 7f521aed4640 1 Processor -- start 2024-04-24T23:49:48.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.606+0000 7f521aed4640 1 -- start start 2024-04-24T23:49:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.607+0000 7f521aed4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f521415dfd0 0x7f521406b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.607+0000 7f521aed4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f52140b8060 0x7f521406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.607+0000 7f521aed4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52140d24d0 con 0x7f521406bc40 2024-04-24T23:49:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.607+0000 7f521aed4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f52140d27b0 con 0x7f521415dfd0 2024-04-24T23:49:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.607+0000 7f521aed4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f52140d2a90 con 0x7f52140b8060 2024-04-24T23:49:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.607+0000 7f5213fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f521406bc40 0x7f52140cefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42246/0 (socket says 172.21.15.16:42246) 2024-04-24T23:49:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.607+0000 7f5213fff640 1 -- 172.21.15.16:0/216247606 learned_addr learned my addr 172.21.15.16:0/216247606 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.607+0000 7f5218c70640 1 --2- 172.21.15.16:0/216247606 >> v2:172.21.15.156:3300/0 conn(0x7f521415dfd0 0x7f521406b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.608+0000 7f5219471640 1 --2- 172.21.15.16:0/216247606 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f52140b8060 0x7f521406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.608+0000 7f52137fe640 1 -- 172.21.15.16:0/216247606 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2420804437 0 0) 0x7f52140d24d0 con 0x7f521406bc40 2024-04-24T23:49:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.608+0000 7f52137fe640 1 -- 172.21.15.16:0/216247606 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51fc003520 con 0x7f521406bc40 2024-04-24T23:49:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.608+0000 7f5218c70640 1 -- 172.21.15.16:0/216247606 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f52140b8060 msgr2=0x7f521406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:48.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.608+0000 7f5218c70640 1 --2- 172.21.15.16:0/216247606 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f52140b8060 0x7f521406c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.608+0000 7f5218c70640 1 -- 172.21.15.16:0/216247606 >> v1:172.21.15.16:6789/0 conn(0x7f521406bc40 legacy=0x7f52140cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.608+0000 7f52137fe640 1 -- 172.21.15.16:0/216247606 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1857089026 0 0) 0x7f51fc003520 con 0x7f521406bc40 2024-04-24T23:49:48.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.608+0000 7f5218c70640 1 -- 172.21.15.16:0/216247606 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52140d2d70 con 0x7f521415dfd0 2024-04-24T23:49:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.609+0000 7f5218c70640 1 --2- 172.21.15.16:0/216247606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521415dfd0 0x7f521406b2f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f520c009d90 tx=0x7f520c02a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5f4567480e25930 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.609+0000 7f52137fe640 1 -- 172.21.15.16:0/216247606 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520c031600 con 0x7f521415dfd0 2024-04-24T23:49:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.609+0000 7f52137fe640 1 -- 172.21.15.16:0/216247606 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f520c0317a0 con 0x7f521415dfd0 2024-04-24T23:49:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.609+0000 7f52137fe640 1 -- 172.21.15.16:0/216247606 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f520c031b50 con 0x7f521415dfd0 2024-04-24T23:49:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.609+0000 7f521aed4640 1 -- 172.21.15.16:0/216247606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521415dfd0 msgr2=0x7f521406b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.609+0000 7f521aed4640 1 --2- 172.21.15.16:0/216247606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521415dfd0 0x7f521406b2f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f520c009d90 tx=0x7f520c02a5d0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.609+0000 7f521aed4640 1 -- 172.21.15.16:0/216247606 shutdown_connections 2024-04-24T23:49:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.609+0000 7f521aed4640 1 --2- 172.21.15.16:0/216247606 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f52140b8060 0x7f521406c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.609+0000 7f521aed4640 1 --2- 172.21.15.16:0/216247606 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521415dfd0 0x7f521406b2f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.609+0000 7f521aed4640 1 -- 172.21.15.16:0/216247606 >> 172.21.15.16:0/216247606 conn(0x7f52140d3dc0 msgr2=0x7f52140d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.609+0000 7f521aed4640 1 -- 172.21.15.16:0/216247606 shutdown_connections 2024-04-24T23:49:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.609+0000 7f521aed4640 1 -- 172.21.15.16:0/216247606 wait complete. 2024-04-24T23:49:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.610+0000 7f521aed4640 1 Processor -- start 2024-04-24T23:49:48.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.611+0000 7f521aed4640 1 -- start start 2024-04-24T23:49:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.611+0000 7f521aed4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f521406bc40 0x7f52140b2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.611+0000 7f521aed4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52140b8060 0x7f52141e81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.611+0000 7f521aed4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521415dfd0 0x7f52140cd4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.611+0000 7f521aed4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f521406b830 con 0x7f52140b8060 2024-04-24T23:49:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.611+0000 7f521aed4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f52140cead0 con 0x7f521415dfd0 2024-04-24T23:49:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.611+0000 7f521aed4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f521406b9b0 con 0x7f521406bc40 2024-04-24T23:49:48.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.611+0000 7f5213fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52140b8060 0x7f52141e81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f5213fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52140b8060 0x7f52141e81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40586/0 (socket says 172.21.15.16:40586) 2024-04-24T23:49:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f5213fff640 1 -- 172.21.15.16:0/3209036265 learned_addr learned my addr 172.21.15.16:0/3209036265 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f5218c70640 1 --2- 172.21.15.16:0/3209036265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f521406bc40 0x7f52140b2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f5219471640 1 --2- 172.21.15.16:0/3209036265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521415dfd0 0x7f52140cd4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f5213fff640 1 -- 172.21.15.16:0/3209036265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f521406bc40 msgr2=0x7f52140b2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f5213fff640 1 --2- 172.21.15.16:0/3209036265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f521406bc40 0x7f52140b2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f5213fff640 1 -- 172.21.15.16:0/3209036265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521415dfd0 msgr2=0x7f52140cd4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f5213fff640 1 --2- 172.21.15.16:0/3209036265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521415dfd0 0x7f52140cd4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f5213fff640 1 -- 172.21.15.16:0/3209036265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52141e89e0 con 0x7f52140b8060 2024-04-24T23:49:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f5213fff640 1 --2- 172.21.15.16:0/3209036265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52140b8060 0x7f52141e81b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f520800c780 tx=0x7f520800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f52117fa640 1 -- 172.21.15.16:0/3209036265 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5208005ab0 con 0x7f52140b8060 2024-04-24T23:49:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f52117fa640 1 -- 172.21.15.16:0/3209036265 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5208005c50 con 0x7f52140b8060 2024-04-24T23:49:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f521aed4640 1 -- 172.21.15.16:0/3209036265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52140caf30 con 0x7f52140b8060 2024-04-24T23:49:48.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f52117fa640 1 -- 172.21.15.16:0/3209036265 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52080072b0 con 0x7f52140b8060 2024-04-24T23:49:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.612+0000 7f521aed4640 1 -- 172.21.15.16:0/3209036265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52141e86f0 con 0x7f52140b8060 2024-04-24T23:49:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.613+0000 7f521aed4640 1 -- 172.21.15.16:0/3209036265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52140cb100 con 0x7f52140b8060 2024-04-24T23:49:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.614+0000 7f52117fa640 1 -- 172.21.15.16:0/3209036265 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5208007450 con 0x7f52140b8060 2024-04-24T23:49:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.614+0000 7f52117fa640 1 --2- 172.21.15.16:0/3209036265 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f520006be30 0x7f520006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.614+0000 7f52117fa640 1 -- 172.21.15.16:0/3209036265 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f520808ad40 con 0x7f52140b8060 2024-04-24T23:49:48.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.614+0000 7f5218c70640 1 --2- 172.21.15.16:0/3209036265 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f520006be30 0x7f520006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.615+0000 7f5218c70640 1 --2- 172.21.15.16:0/3209036265 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f520006be30 0x7f520006e2f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f520c034000 tx=0x7f520c02f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:48.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.618+0000 7f52117fa640 1 -- 172.21.15.16:0/3209036265 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5208058d30 con 0x7f52140b8060 2024-04-24T23:49:48.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.801+0000 7f521aed4640 1 -- 172.21.15.16:0/3209036265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5214068f90 con 0x7f52140b8060 2024-04-24T23:49:48.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.801+0000 7f52117fa640 1 -- 172.21.15.16:0/3209036265 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f520805d030 con 0x7f52140b8060 2024-04-24T23:49:48.801 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.803+0000 7f521aed4640 1 -- 172.21.15.16:0/3209036265 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f520006be30 msgr2=0x7f520006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.803+0000 7f521aed4640 1 --2- 172.21.15.16:0/3209036265 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f520006be30 0x7f520006e2f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f520c034000 tx=0x7f520c02f000 comp rx=0 tx=0).stop 2024-04-24T23:49:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.803+0000 7f521aed4640 1 -- 172.21.15.16:0/3209036265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52140b8060 msgr2=0x7f52141e81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.803+0000 7f521aed4640 1 --2- 172.21.15.16:0/3209036265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52140b8060 0x7f52141e81b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f520800c780 tx=0x7f520800cc40 comp rx=0 tx=0).stop 2024-04-24T23:49:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.803+0000 7f521aed4640 1 -- 172.21.15.16:0/3209036265 shutdown_connections 2024-04-24T23:49:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.803+0000 7f521aed4640 1 --2- 172.21.15.16:0/3209036265 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f520006be30 0x7f520006e2f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.803+0000 7f521aed4640 1 --2- 172.21.15.16:0/3209036265 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f521415dfd0 0x7f52140cd4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.803+0000 7f521aed4640 1 --2- 172.21.15.16:0/3209036265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f52140b8060 0x7f52141e81b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.803+0000 7f521aed4640 1 --2- 172.21.15.16:0/3209036265 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f521406bc40 0x7f52140b2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.803+0000 7f521aed4640 1 -- 172.21.15.16:0/3209036265 >> 172.21.15.16:0/3209036265 conn(0x7f52140d3dc0 msgr2=0x7f52140b48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:48.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.803+0000 7f521aed4640 1 -- 172.21.15.16:0/3209036265 shutdown_connections 2024-04-24T23:49:48.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.803+0000 7f521aed4640 1 -- 172.21.15.16:0/3209036265 wait complete. 2024-04-24T23:49:48.815 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:48.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.918+0000 7fc455d9d640 1 Processor -- start 2024-04-24T23:49:48.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.919+0000 7fc455d9d640 1 -- start start 2024-04-24T23:49:48.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.919+0000 7fc455d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc45006c0d0 0x7fc4500b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.919+0000 7fc455d9d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4500d24d0 con 0x7fc45006bc40 2024-04-24T23:49:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.919+0000 7fc455d9d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4500d27b0 con 0x7fc45015dfd0 2024-04-24T23:49:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.919+0000 7fc455d9d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fc45006ba00 con 0x7fc45006c0d0 2024-04-24T23:49:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.919+0000 7fc44effd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc45006bc40 0x7fc4500b7050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42260/0 (socket says 172.21.15.16:42260) 2024-04-24T23:49:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.919+0000 7fc44ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc45006c0d0 0x7fc4500b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.919+0000 7fc44effd640 1 -- 172.21.15.16:0/3157661332 learned_addr learned my addr 172.21.15.16:0/3157661332 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.920+0000 7fc44ffff640 1 -- 172.21.15.16:0/3157661332 >> v1:172.21.15.156:6789/0 conn(0x7fc45015dfd0 legacy=0x7fc45006b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.920+0000 7fc44e7fc640 1 -- 172.21.15.16:0/3157661332 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 858853084 0 0) 0x7fc4500d24d0 con 0x7fc45006bc40 2024-04-24T23:49:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.920+0000 7fc44ffff640 1 -- 172.21.15.16:0/3157661332 >> v1:172.21.15.16:6789/0 conn(0x7fc45006bc40 legacy=0x7fc4500b7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.920+0000 7fc44ffff640 1 -- 172.21.15.16:0/3157661332 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4500d2a90 con 0x7fc45006c0d0 2024-04-24T23:49:48.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.921+0000 7fc44ffff640 1 --2- 172.21.15.16:0/3157661332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45006c0d0 0x7fc4500b7760 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc448009d90 tx=0x7fc44802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2141bb1a3fbf08ed server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:48.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.921+0000 7fc44e7fc640 1 -- 172.21.15.16:0/3157661332 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc448007740 con 0x7fc45006c0d0 2024-04-24T23:49:48.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.921+0000 7fc44e7fc640 1 -- 172.21.15.16:0/3157661332 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4480078e0 con 0x7fc45006c0d0 2024-04-24T23:49:48.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.921+0000 7fc44e7fc640 1 -- 172.21.15.16:0/3157661332 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc448007cb0 con 0x7fc45006c0d0 2024-04-24T23:49:48.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.921+0000 7fc455d9d640 1 -- 172.21.15.16:0/3157661332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45006c0d0 msgr2=0x7fc4500b7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.921+0000 7fc455d9d640 1 --2- 172.21.15.16:0/3157661332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45006c0d0 0x7fc4500b7760 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc448009d90 tx=0x7fc44802a660 comp rx=0 tx=0).stop 2024-04-24T23:49:48.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.921+0000 7fc455d9d640 1 -- 172.21.15.16:0/3157661332 shutdown_connections 2024-04-24T23:49:48.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.921+0000 7fc455d9d640 1 --2- 172.21.15.16:0/3157661332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45006c0d0 0x7fc4500b7760 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.921+0000 7fc455d9d640 1 -- 172.21.15.16:0/3157661332 >> 172.21.15.16:0/3157661332 conn(0x7fc4500d3dc0 msgr2=0x7fc4500d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:48.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.921+0000 7fc455d9d640 1 -- 172.21.15.16:0/3157661332 shutdown_connections 2024-04-24T23:49:48.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.921+0000 7fc455d9d640 1 -- 172.21.15.16:0/3157661332 wait complete. 2024-04-24T23:49:48.924 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.922+0000 7fc455d9d640 1 Processor -- start 2024-04-24T23:49:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.923+0000 7fc455d9d640 1 -- start start 2024-04-24T23:49:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.923+0000 7fc455d9d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc45006bc40 0x7fc450159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.923+0000 7fc455d9d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc45006c0d0 0x7fc4500cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.923+0000 7fc455d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45015dfd0 0x7fc45015b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.923+0000 7fc455d9d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc45006c4e0 con 0x7fc45015dfd0 2024-04-24T23:49:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.923+0000 7fc455d9d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc4500d3050 con 0x7fc45006c0d0 2024-04-24T23:49:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.923+0000 7fc455d9d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fc4500cead0 con 0x7fc45006bc40 2024-04-24T23:49:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.923+0000 7fc44ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45015dfd0 0x7fc45015b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc44ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45015dfd0 0x7fc45015b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40610/0 (socket says 172.21.15.16:40610) 2024-04-24T23:49:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc44ffff640 1 -- 172.21.15.16:0/3944925831 learned_addr learned my addr 172.21.15.16:0/3944925831 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc44effd640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc45006c0d0 0x7fc4500cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc44f7fe640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc45006bc40 0x7fc450159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc44ffff640 1 -- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc45006bc40 msgr2=0x7fc450159690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc44ffff640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc45006bc40 0x7fc450159690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc44ffff640 1 -- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc45006c0d0 msgr2=0x7fc4500cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:48.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc44ffff640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc45006c0d0 0x7fc4500cd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:48.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc44ffff640 1 -- 172.21.15.16:0/3944925831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc450159d80 con 0x7fc45015dfd0 2024-04-24T23:49:48.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc44effd640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc45006c0d0 0x7fc4500cd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:48.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc44ffff640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45015dfd0 0x7fc45015b7f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc448009eb0 tx=0x7fc448004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:48.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc437fff640 1 -- 172.21.15.16:0/3944925831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc448040070 con 0x7fc45015dfd0 2024-04-24T23:49:48.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc437fff640 1 -- 172.21.15.16:0/3944925831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc448031c90 con 0x7fc45015dfd0 2024-04-24T23:49:48.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc455d9d640 1 -- 172.21.15.16:0/3944925831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc45015a060 con 0x7fc45015dfd0 2024-04-24T23:49:48.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc437fff640 1 -- 172.21.15.16:0/3944925831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc44803c640 con 0x7fc45015dfd0 2024-04-24T23:49:48.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.924+0000 7fc455d9d640 1 -- 172.21.15.16:0/3944925831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4500cafc0 con 0x7fc45015dfd0 2024-04-24T23:49:48.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.925+0000 7fc455d9d640 1 -- 172.21.15.16:0/3944925831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4500cb1c0 con 0x7fc45015dfd0 2024-04-24T23:49:48.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.926+0000 7fc437fff640 1 -- 172.21.15.16:0/3944925831 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc448039040 con 0x7fc45015dfd0 2024-04-24T23:49:48.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.926+0000 7fc437fff640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc43c06be30 0x7fc43c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:48.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.926+0000 7fc437fff640 1 -- 172.21.15.16:0/3944925831 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fc4480ac840 con 0x7fc45015dfd0 2024-04-24T23:49:48.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.927+0000 7fc44f7fe640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc43c06be30 0x7fc43c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:48.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.927+0000 7fc44f7fe640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc43c06be30 0x7fc43c06e2f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc440001580 tx=0x7fc440009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:48.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:48.930+0000 7fc437fff640 1 -- 172.21.15.16:0/3944925831 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc44807af40 con 0x7fc45015dfd0 2024-04-24T23:49:49.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.111+0000 7fc455d9d640 1 -- 172.21.15.16:0/3944925831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc450068f90 con 0x7fc45015dfd0 2024-04-24T23:49:49.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.111+0000 7fc437fff640 1 -- 172.21.15.16:0/3944925831 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fc44807a840 con 0x7fc45015dfd0 2024-04-24T23:49:49.111 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:49.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.113+0000 7fc455d9d640 1 -- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc43c06be30 msgr2=0x7fc43c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.113+0000 7fc455d9d640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc43c06be30 0x7fc43c06e2f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc440001580 tx=0x7fc440009040 comp rx=0 tx=0).stop 2024-04-24T23:49:49.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.113+0000 7fc455d9d640 1 -- 172.21.15.16:0/3944925831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45015dfd0 msgr2=0x7fc45015b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.113+0000 7fc455d9d640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45015dfd0 0x7fc45015b7f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc448009eb0 tx=0x7fc448004a40 comp rx=0 tx=0).stop 2024-04-24T23:49:49.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.113+0000 7fc455d9d640 1 -- 172.21.15.16:0/3944925831 shutdown_connections 2024-04-24T23:49:49.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.113+0000 7fc455d9d640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc43c06be30 0x7fc43c06e2f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.113+0000 7fc455d9d640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc45015dfd0 0x7fc45015b7f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.113+0000 7fc455d9d640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc45006c0d0 0x7fc4500cd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.113+0000 7fc455d9d640 1 --2- 172.21.15.16:0/3944925831 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc45006bc40 0x7fc450159690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.113+0000 7fc455d9d640 1 -- 172.21.15.16:0/3944925831 >> 172.21.15.16:0/3944925831 conn(0x7fc4500d3dc0 msgr2=0x7fc45006b2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:49.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.114+0000 7fc455d9d640 1 -- 172.21.15.16:0/3944925831 shutdown_connections 2024-04-24T23:49:49.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.114+0000 7fc455d9d640 1 -- 172.21.15.16:0/3944925831 wait complete. 2024-04-24T23:49:49.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:49.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.228+0000 7f88aa441640 1 Processor -- start 2024-04-24T23:49:49.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.228+0000 7f88aa441640 1 -- start start 2024-04-24T23:49:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.229+0000 7f88aa441640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f88a415dfd0 0x7f88a406b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.229+0000 7f88aa441640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88a40b8060 0x7f88a406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.229+0000 7f88aa441640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88a40d24d0 con 0x7f88a406bc40 2024-04-24T23:49:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.229+0000 7f88aa441640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f88a40d27b0 con 0x7f88a415dfd0 2024-04-24T23:49:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.229+0000 7f88aa441640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f88a40d2a90 con 0x7f88a40b8060 2024-04-24T23:49:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.229+0000 7f88a3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f88a415dfd0 0x7f88a406b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.229+0000 7f88a3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f88a415dfd0 0x7f88a406b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40622/0 (socket says 172.21.15.16:40622) 2024-04-24T23:49:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.229+0000 7f88a3fff640 1 -- 172.21.15.16:0/3494339500 learned_addr learned my addr 172.21.15.16:0/3494339500 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.230+0000 7f88a3fff640 1 -- 172.21.15.16:0/3494339500 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88a40b8060 msgr2=0x7f88a406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.230+0000 7f88a89de640 1 --2- 172.21.15.16:0/3494339500 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88a40b8060 0x7f88a406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.230+0000 7f88a3fff640 1 --2- 172.21.15.16:0/3494339500 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88a40b8060 0x7f88a406c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.230+0000 7f88a3fff640 1 -- 172.21.15.16:0/3494339500 >> v1:172.21.15.156:6789/0 conn(0x7f88a406bc40 legacy=0x7f88a40cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:49.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.230+0000 7f88a3fff640 1 -- 172.21.15.16:0/3494339500 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a40d2d70 con 0x7f88a415dfd0 2024-04-24T23:49:49.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.230+0000 7f88a89de640 1 --2- 172.21.15.16:0/3494339500 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88a40b8060 0x7f88a406c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:49.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.230+0000 7f88a3fff640 1 --2- 172.21.15.16:0/3494339500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a415dfd0 0x7f88a406b2f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f888c009870 tx=0x7f888c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b3cd80ea24cc50 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.231+0000 7f88a2ffd640 1 -- 172.21.15.16:0/3494339500 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c037070 con 0x7f88a415dfd0 2024-04-24T23:49:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.231+0000 7f88a2ffd640 1 -- 172.21.15.16:0/3494339500 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f888c032440 con 0x7f88a415dfd0 2024-04-24T23:49:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.231+0000 7f88a2ffd640 1 -- 172.21.15.16:0/3494339500 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c032810 con 0x7f88a415dfd0 2024-04-24T23:49:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.231+0000 7f88aa441640 1 -- 172.21.15.16:0/3494339500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a415dfd0 msgr2=0x7f88a406b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.231+0000 7f88aa441640 1 --2- 172.21.15.16:0/3494339500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a415dfd0 0x7f88a406b2f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f888c009870 tx=0x7f888c02a180 comp rx=0 tx=0).stop 2024-04-24T23:49:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.231+0000 7f88aa441640 1 -- 172.21.15.16:0/3494339500 shutdown_connections 2024-04-24T23:49:49.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.231+0000 7f88aa441640 1 --2- 172.21.15.16:0/3494339500 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88a40b8060 0x7f88a406c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.231+0000 7f88aa441640 1 --2- 172.21.15.16:0/3494339500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a415dfd0 0x7f88a406b2f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.231+0000 7f88aa441640 1 -- 172.21.15.16:0/3494339500 >> 172.21.15.16:0/3494339500 conn(0x7f88a40d3dc0 msgr2=0x7f88a40d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.231+0000 7f88aa441640 1 -- 172.21.15.16:0/3494339500 shutdown_connections 2024-04-24T23:49:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.231+0000 7f88aa441640 1 -- 172.21.15.16:0/3494339500 wait complete. 2024-04-24T23:49:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.232+0000 7f88aa441640 1 Processor -- start 2024-04-24T23:49:49.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.232+0000 7f88aa441640 1 -- start start 2024-04-24T23:49:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.233+0000 7f88aa441640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88a406bc40 0x7f88a40c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.233+0000 7f88aa441640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a40b8060 0x7f88a415ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.233+0000 7f88aa441640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88a415dfd0 0x7f88a414bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.233+0000 7f88aa441640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f88a406b830 con 0x7f88a40b8060 2024-04-24T23:49:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.233+0000 7f88aa441640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f88a40cead0 con 0x7f88a415dfd0 2024-04-24T23:49:49.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.233+0000 7f88aa441640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f88a406b9b0 con 0x7f88a406bc40 2024-04-24T23:49:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.233+0000 7f88a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a40b8060 0x7f88a415ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.233+0000 7f88a37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a40b8060 0x7f88a415ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40624/0 (socket says 172.21.15.16:40624) 2024-04-24T23:49:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.233+0000 7f88a37fe640 1 -- 172.21.15.16:0/2999860663 learned_addr learned my addr 172.21.15.16:0/2999860663 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.233+0000 7f88a3fff640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88a406bc40 0x7f88a40c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.233+0000 7f88a89de640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88a415dfd0 0x7f88a414bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.233+0000 7f88a37fe640 1 -- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88a406bc40 msgr2=0x7f88a40c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.233+0000 7f88a37fe640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88a406bc40 0x7f88a40c98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.234+0000 7f88a37fe640 1 -- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88a415dfd0 msgr2=0x7f88a414bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.234+0000 7f88a37fe640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88a415dfd0 0x7f88a414bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.234+0000 7f88a37fe640 1 -- 172.21.15.16:0/2999860663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88a4160a00 con 0x7f88a40b8060 2024-04-24T23:49:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.234+0000 7f88a3fff640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88a406bc40 0x7f88a40c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:49.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.234+0000 7f88a89de640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88a415dfd0 0x7f88a414bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.234+0000 7f88a37fe640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a40b8060 0x7f88a415ff10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f889400c430 tx=0x7f889400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.234+0000 7f88a0ff9640 1 -- 172.21.15.16:0/2999860663 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894011790 con 0x7f88a40b8060 2024-04-24T23:49:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.234+0000 7f88a0ff9640 1 -- 172.21.15.16:0/2999860663 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8894017070 con 0x7f88a40b8060 2024-04-24T23:49:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.234+0000 7f88aa441640 1 -- 172.21.15.16:0/2999860663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88a4160ce0 con 0x7f88a40b8060 2024-04-24T23:49:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.234+0000 7f88a0ff9640 1 -- 172.21.15.16:0/2999860663 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894011bb0 con 0x7f88a40b8060 2024-04-24T23:49:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.234+0000 7f88aa441640 1 -- 172.21.15.16:0/2999860663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88a414a180 con 0x7f88a40b8060 2024-04-24T23:49:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.235+0000 7f88aa441640 1 -- 172.21.15.16:0/2999860663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88a4149c90 con 0x7f88a40b8060 2024-04-24T23:49:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.235+0000 7f88a0ff9640 1 -- 172.21.15.16:0/2999860663 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f889401c460 con 0x7f88a40b8060 2024-04-24T23:49:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.236+0000 7f88a0ff9640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f888406be30 0x7f888406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.236+0000 7f88a0ff9640 1 -- 172.21.15.16:0/2999860663 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f889408b200 con 0x7f88a40b8060 2024-04-24T23:49:49.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.236+0000 7f88a3fff640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f888406be30 0x7f888406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.236+0000 7f88a3fff640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f888406be30 0x7f888406e2f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f888c034000 tx=0x7f888c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:49.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.240+0000 7f88a0ff9640 1 -- 172.21.15.16:0/2999860663 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8894059710 con 0x7f88a40b8060 2024-04-24T23:49:49.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.423+0000 7f88aa441640 1 -- 172.21.15.16:0/2999860663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f88a40d0340 con 0x7f88a40b8060 2024-04-24T23:49:49.423 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.424+0000 7f88a0ff9640 1 -- 172.21.15.16:0/2999860663 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f8894059010 con 0x7f88a40b8060 2024-04-24T23:49:49.423 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.425+0000 7f88aa441640 1 -- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f888406be30 msgr2=0x7f888406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.425+0000 7f88aa441640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f888406be30 0x7f888406e2f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f888c034000 tx=0x7f888c02e000 comp rx=0 tx=0).stop 2024-04-24T23:49:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.425+0000 7f88aa441640 1 -- 172.21.15.16:0/2999860663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a40b8060 msgr2=0x7f88a415ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.425+0000 7f88aa441640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a40b8060 0x7f88a415ff10 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f889400c430 tx=0x7f889400c8f0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.425+0000 7f88aa441640 1 -- 172.21.15.16:0/2999860663 shutdown_connections 2024-04-24T23:49:49.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.426+0000 7f88aa441640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f888406be30 0x7f888406e2f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.426+0000 7f88aa441640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f88a415dfd0 0x7f88a414bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.426+0000 7f88aa441640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f88a40b8060 0x7f88a415ff10 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.426+0000 7f88aa441640 1 --2- 172.21.15.16:0/2999860663 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f88a406bc40 0x7f88a40c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.426+0000 7f88aa441640 1 -- 172.21.15.16:0/2999860663 >> 172.21.15.16:0/2999860663 conn(0x7f88a40d3dc0 msgr2=0x7f88a40ccea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:49.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.426+0000 7f88aa441640 1 -- 172.21.15.16:0/2999860663 shutdown_connections 2024-04-24T23:49:49.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.426+0000 7f88aa441640 1 -- 172.21.15.16:0/2999860663 wait complete. 2024-04-24T23:49:49.436 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:49.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.540+0000 7f2420363640 1 Processor -- start 2024-04-24T23:49:49.540 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.540+0000 7f2420363640 1 -- start start 2024-04-24T23:49:49.541 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.541+0000 7f2420363640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f241806c080 0x7f24180ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.541+0000 7f2420363640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24180aebb0 con 0x7f2418161510 2024-04-24T23:49:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.541+0000 7f2420363640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24180adfa0 con 0x7f24180b6230 2024-04-24T23:49:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.541+0000 7f2420363640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f24180ae280 con 0x7f241806c080 2024-04-24T23:49:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.542+0000 7f241d8fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f241806c080 0x7f24180ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.542+0000 7f241e0ff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2418161510 0x7f241806b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42276/0 (socket says 172.21.15.16:42276) 2024-04-24T23:49:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.542+0000 7f241e0ff640 1 -- 172.21.15.16:0/1115827357 learned_addr learned my addr 172.21.15.16:0/1115827357 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.542+0000 7f241d0fd640 1 -- 172.21.15.16:0/1115827357 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2288575651 0 0) 0x7f24180aebb0 con 0x7f2418161510 2024-04-24T23:49:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.542+0000 7f241d0fd640 1 -- 172.21.15.16:0/1115827357 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2404003520 con 0x7f2418161510 2024-04-24T23:49:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.542+0000 7f241d8fe640 1 -- 172.21.15.16:0/1115827357 >> v1:172.21.15.156:6789/0 conn(0x7f24180b6230 legacy=0x7f24180b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.542+0000 7f241d8fe640 1 -- 172.21.15.16:0/1115827357 >> v1:172.21.15.16:6789/0 conn(0x7f2418161510 legacy=0x7f241806b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.542+0000 7f241d8fe640 1 -- 172.21.15.16:0/1115827357 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24180ae560 con 0x7f241806c080 2024-04-24T23:49:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.543+0000 7f241d8fe640 1 --2- 172.21.15.16:0/1115827357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f241806c080 0x7f24180ad8e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f24080097a0 tx=0x7f240802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccdf2ecf6d7b191e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.543+0000 7f241d0fd640 1 -- 172.21.15.16:0/1115827357 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2408037070 con 0x7f241806c080 2024-04-24T23:49:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.543+0000 7f241d0fd640 1 -- 172.21.15.16:0/1115827357 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2408032440 con 0x7f241806c080 2024-04-24T23:49:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.543+0000 7f241d0fd640 1 -- 172.21.15.16:0/1115827357 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2408032810 con 0x7f241806c080 2024-04-24T23:49:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.543+0000 7f2420363640 1 -- 172.21.15.16:0/1115827357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f241806c080 msgr2=0x7f24180ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.543+0000 7f2420363640 1 --2- 172.21.15.16:0/1115827357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f241806c080 0x7f24180ad8e0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f24080097a0 tx=0x7f240802a1e0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.543+0000 7f2420363640 1 -- 172.21.15.16:0/1115827357 shutdown_connections 2024-04-24T23:49:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.543+0000 7f2420363640 1 --2- 172.21.15.16:0/1115827357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f241806c080 0x7f24180ad8e0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.543+0000 7f2420363640 1 -- 172.21.15.16:0/1115827357 >> 172.21.15.16:0/1115827357 conn(0x7f2418072aa0 msgr2=0x7f2418072eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.544+0000 7f2420363640 1 -- 172.21.15.16:0/1115827357 shutdown_connections 2024-04-24T23:49:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.544+0000 7f2420363640 1 -- 172.21.15.16:0/1115827357 wait complete. 2024-04-24T23:49:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.545+0000 7f2420363640 1 Processor -- start 2024-04-24T23:49:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.545+0000 7f2420363640 1 -- start start 2024-04-24T23:49:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.545+0000 7f2420363640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f241806c080 0x7f2418158460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.545+0000 7f2420363640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24180b6230 0x7f24181558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.545+0000 7f2420363640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2418161510 0x7f24181410a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.545+0000 7f2420363640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f241806da90 con 0x7f2418161510 2024-04-24T23:49:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.545+0000 7f2420363640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f24180b6660 con 0x7f24180b6230 2024-04-24T23:49:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f2420363640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f241806dc10 con 0x7f241806c080 2024-04-24T23:49:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f241e900640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2418161510 0x7f24181410a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f241e900640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2418161510 0x7f24181410a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40644/0 (socket says 172.21.15.16:40644) 2024-04-24T23:49:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f241e900640 1 -- 172.21.15.16:0/2241895926 learned_addr learned my addr 172.21.15.16:0/2241895926 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f241d8fe640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24180b6230 0x7f24181558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f241e0ff640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f241806c080 0x7f2418158460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f241e900640 1 -- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f241806c080 msgr2=0x7f2418158460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f241e900640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f241806c080 0x7f2418158460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f241e900640 1 -- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24180b6230 msgr2=0x7f24181558f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f241e900640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24180b6230 0x7f24181558f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f241e900640 1 -- 172.21.15.16:0/2241895926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2418155e30 con 0x7f2418161510 2024-04-24T23:49:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f241d8fe640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24180b6230 0x7f24181558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f241e0ff640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f241806c080 0x7f2418158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f241e900640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2418161510 0x7f24181410a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f241000c780 tx=0x7f241000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f240effd640 1 -- 172.21.15.16:0/2241895926 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2410004b60 con 0x7f2418161510 2024-04-24T23:49:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f240effd640 1 -- 172.21.15.16:0/2241895926 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2410004d00 con 0x7f2418161510 2024-04-24T23:49:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f2420363640 1 -- 172.21.15.16:0/2241895926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2418158b40 con 0x7f2418161510 2024-04-24T23:49:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f240effd640 1 -- 172.21.15.16:0/2241895926 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241000aa90 con 0x7f2418161510 2024-04-24T23:49:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.546+0000 7f2420363640 1 -- 172.21.15.16:0/2241895926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2418159100 con 0x7f2418161510 2024-04-24T23:49:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.547+0000 7f2420363640 1 -- 172.21.15.16:0/2241895926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2418068f90 con 0x7f2418161510 2024-04-24T23:49:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.548+0000 7f240effd640 1 -- 172.21.15.16:0/2241895926 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f241000ac30 con 0x7f2418161510 2024-04-24T23:49:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.548+0000 7f240effd640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f23fc06be30 0x7f23fc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.548+0000 7f240effd640 1 -- 172.21.15.16:0/2241895926 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f241008b4f0 con 0x7f2418161510 2024-04-24T23:49:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.549+0000 7f241e0ff640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f23fc06be30 0x7f23fc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.549+0000 7f241e0ff640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f23fc06be30 0x7f23fc06e2f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2414000f60 tx=0x7f2414009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.552+0000 7f240effd640 1 -- 172.21.15.16:0/2241895926 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2410020070 con 0x7f2418161510 2024-04-24T23:49:49.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.735+0000 7f2420363640 1 -- 172.21.15.16:0/2241895926 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f241806c860 con 0x7f2418161510 2024-04-24T23:49:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.735+0000 7f240effd640 1 -- 172.21.15.16:0/2241895926 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f2410059aa0 con 0x7f2418161510 2024-04-24T23:49:49.735 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.737+0000 7f2420363640 1 -- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f23fc06be30 msgr2=0x7f23fc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.737+0000 7f2420363640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f23fc06be30 0x7f23fc06e2f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2414000f60 tx=0x7f2414009040 comp rx=0 tx=0).stop 2024-04-24T23:49:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.737+0000 7f2420363640 1 -- 172.21.15.16:0/2241895926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2418161510 msgr2=0x7f24181410a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.737+0000 7f2420363640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2418161510 0x7f24181410a0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f241000c780 tx=0x7f241000cc40 comp rx=0 tx=0).stop 2024-04-24T23:49:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.737+0000 7f2420363640 1 -- 172.21.15.16:0/2241895926 shutdown_connections 2024-04-24T23:49:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.737+0000 7f2420363640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f23fc06be30 0x7f23fc06e2f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.737+0000 7f2420363640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2418161510 0x7f24181410a0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.737+0000 7f2420363640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24180b6230 0x7f24181558f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.737+0000 7f2420363640 1 --2- 172.21.15.16:0/2241895926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f241806c080 0x7f2418158460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.737+0000 7f2420363640 1 -- 172.21.15.16:0/2241895926 >> 172.21.15.16:0/2241895926 conn(0x7f2418072aa0 msgr2=0x7f2418161950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:49.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.737+0000 7f2420363640 1 -- 172.21.15.16:0/2241895926 shutdown_connections 2024-04-24T23:49:49.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.737+0000 7f2420363640 1 -- 172.21.15.16:0/2241895926 wait complete. 2024-04-24T23:49:49.748 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:49:49.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.850+0000 7f2eb23b7640 1 Processor -- start 2024-04-24T23:49:49.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.850+0000 7f2eb23b7640 1 -- start start 2024-04-24T23:49:49.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.851+0000 7f2eb23b7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eac071a00 0x7f2eac071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.851+0000 7f2eb23b7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2eac144740 0x7f2eac15c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.851+0000 7f2eb23b7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eac072320 con 0x7f2eac15c8e0 2024-04-24T23:49:49.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.851+0000 7f2eb23b7640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2eac15c1f0 con 0x7f2eac144740 2024-04-24T23:49:49.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.851+0000 7f2eb23b7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2eac0cf3a0 con 0x7f2eac071a00 2024-04-24T23:49:49.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.851+0000 7f2eab7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2eac144740 0x7f2eac15c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.851+0000 7f2eb0954640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2eac15c8e0 0x7f2eac0ce7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42280/0 (socket says 172.21.15.16:42280) 2024-04-24T23:49:49.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.851+0000 7f2eb0954640 1 -- 172.21.15.16:0/2127898487 learned_addr learned my addr 172.21.15.16:0/2127898487 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:49.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.851+0000 7f2eabfff640 1 --2- 172.21.15.16:0/2127898487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eac071a00 0x7f2eac071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.851+0000 7f2eab7fe640 1 -- 172.21.15.16:0/2127898487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eac071a00 msgr2=0x7f2eac071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.851+0000 7f2eab7fe640 1 --2- 172.21.15.16:0/2127898487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eac071a00 0x7f2eac071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.851+0000 7f2eab7fe640 1 -- 172.21.15.16:0/2127898487 >> v1:172.21.15.16:6789/0 conn(0x7f2eac15c8e0 legacy=0x7f2eac0ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.851+0000 7f2eab7fe640 1 -- 172.21.15.16:0/2127898487 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac0cc8d0 con 0x7f2eac144740 2024-04-24T23:49:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.851+0000 7f2eaaffd640 1 -- 172.21.15.16:0/2127898487 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2270823158 0 0) 0x7f2eac072320 con 0x7f2eac15c8e0 2024-04-24T23:49:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.852+0000 7f2eab7fe640 1 --2- 172.21.15.16:0/2127898487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac144740 0x7f2eac15c3a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0097a0 tx=0x7f2e9c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7aab00dcf708593 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.852+0000 7f2eaaffd640 1 -- 172.21.15.16:0/2127898487 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c037070 con 0x7f2eac144740 2024-04-24T23:49:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.852+0000 7f2eaaffd640 1 -- 172.21.15.16:0/2127898487 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e9c032440 con 0x7f2eac144740 2024-04-24T23:49:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.852+0000 7f2eaaffd640 1 -- 172.21.15.16:0/2127898487 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c032810 con 0x7f2eac144740 2024-04-24T23:49:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.853+0000 7f2eb23b7640 1 -- 172.21.15.16:0/2127898487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac144740 msgr2=0x7f2eac15c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.853+0000 7f2eb23b7640 1 --2- 172.21.15.16:0/2127898487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac144740 0x7f2eac15c3a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0097a0 tx=0x7f2e9c02a180 comp rx=0 tx=0).stop 2024-04-24T23:49:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.853+0000 7f2eb23b7640 1 -- 172.21.15.16:0/2127898487 shutdown_connections 2024-04-24T23:49:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.853+0000 7f2eb23b7640 1 --2- 172.21.15.16:0/2127898487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac144740 0x7f2eac15c3a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.853+0000 7f2eb23b7640 1 --2- 172.21.15.16:0/2127898487 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eac071a00 0x7f2eac071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.853+0000 7f2eb23b7640 1 -- 172.21.15.16:0/2127898487 >> 172.21.15.16:0/2127898487 conn(0x7f2eac0b2000 msgr2=0x7f2eac0cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:49.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.853+0000 7f2eb23b7640 1 -- 172.21.15.16:0/2127898487 shutdown_connections 2024-04-24T23:49:49.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.853+0000 7f2eb23b7640 1 -- 172.21.15.16:0/2127898487 wait complete. 2024-04-24T23:49:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.854+0000 7f2eb23b7640 1 Processor -- start 2024-04-24T23:49:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.854+0000 7f2eb23b7640 1 -- start start 2024-04-24T23:49:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eb23b7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2eac071a00 0x7f2eac0b3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eb23b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac15c8e0 0x7f2eac0ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eb23b7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eac0ace00 0x7f2eac15a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eb23b7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac0cd170 con 0x7f2eac15c8e0 2024-04-24T23:49:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eb23b7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac0cce90 con 0x7f2eac071a00 2024-04-24T23:49:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eb23b7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2eac0ceed0 con 0x7f2eac0ace00 2024-04-24T23:49:49.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac15c8e0 0x7f2eac0ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eab7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac15c8e0 0x7f2eac0ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40664/0 (socket says 172.21.15.16:40664) 2024-04-24T23:49:49.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eab7fe640 1 -- 172.21.15.16:0/1146496191 learned_addr learned my addr 172.21.15.16:0/1146496191 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:49.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eb0954640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eac0ace00 0x7f2eac15a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eabfff640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2eac071a00 0x7f2eac0b3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eab7fe640 1 -- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eac0ace00 msgr2=0x7f2eac15a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eab7fe640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eac0ace00 0x7f2eac15a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eab7fe640 1 -- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2eac071a00 msgr2=0x7f2eac0b3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:49.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eab7fe640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2eac071a00 0x7f2eac0b3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:49.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eab7fe640 1 -- 172.21.15.16:0/1146496191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac0ad380 con 0x7f2eac15c8e0 2024-04-24T23:49:49.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.855+0000 7f2eb0954640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eac0ace00 0x7f2eac15a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:49.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.856+0000 7f2eab7fe640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac15c8e0 0x7f2eac0ac8c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c000c00 tx=0x7f2e9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:49.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.856+0000 7f2ea8ff9640 1 -- 172.21.15.16:0/1146496191 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c037050 con 0x7f2eac15c8e0 2024-04-24T23:49:49.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.856+0000 7f2ea8ff9640 1 -- 172.21.15.16:0/1146496191 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e9c038040 con 0x7f2eac15c8e0 2024-04-24T23:49:49.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.856+0000 7f2eb23b7640 1 -- 172.21.15.16:0/1146496191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eac0ad660 con 0x7f2eac15c8e0 2024-04-24T23:49:49.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.856+0000 7f2ea8ff9640 1 -- 172.21.15.16:0/1146496191 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c004490 con 0x7f2eac15c8e0 2024-04-24T23:49:49.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.856+0000 7f2eb23b7640 1 -- 172.21.15.16:0/1146496191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eac0adc20 con 0x7f2eac15c8e0 2024-04-24T23:49:49.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.857+0000 7f2eb23b7640 1 -- 172.21.15.16:0/1146496191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eac0731f0 con 0x7f2eac15c8e0 2024-04-24T23:49:49.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.858+0000 7f2ea8ff9640 1 -- 172.21.15.16:0/1146496191 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2e9c030900 con 0x7f2eac15c8e0 2024-04-24T23:49:49.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.858+0000 7f2ea8ff9640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e9806be60 0x7f2e9806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.858+0000 7f2ea8ff9640 1 -- 172.21.15.16:0/1146496191 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f2e9c0abef0 con 0x7f2eac15c8e0 2024-04-24T23:49:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.858+0000 7f2eabfff640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e9806be60 0x7f2e9806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:49.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.859+0000 7f2eabfff640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e9806be60 0x7f2e9806e320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0006fd0 tx=0x7f2ea0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:49.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:49.863+0000 7f2ea8ff9640 1 -- 172.21.15.16:0/1146496191 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2e9c07a530 con 0x7f2eac15c8e0 2024-04-24T23:49:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.059+0000 7f2eb23b7640 1 -- 172.21.15.16:0/1146496191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2eac04f5e0 con 0x7f2eac15c8e0 2024-04-24T23:49:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.060+0000 7f2ea8ff9640 1 -- 172.21.15.16:0/1146496191 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f2e9c079e30 con 0x7f2eac15c8e0 2024-04-24T23:49:50.059 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-04-24T23:49:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.062+0000 7f2eb23b7640 1 -- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e9806be60 msgr2=0x7f2e9806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.062+0000 7f2eb23b7640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e9806be60 0x7f2e9806e320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0006fd0 tx=0x7f2ea0008040 comp rx=0 tx=0).stop 2024-04-24T23:49:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.062+0000 7f2eb23b7640 1 -- 172.21.15.16:0/1146496191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac15c8e0 msgr2=0x7f2eac0ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.062+0000 7f2eb23b7640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac15c8e0 0x7f2eac0ac8c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c000c00 tx=0x7f2e9c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.062+0000 7f2eb23b7640 1 -- 172.21.15.16:0/1146496191 shutdown_connections 2024-04-24T23:49:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.062+0000 7f2eb23b7640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e9806be60 0x7f2e9806e320 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.062+0000 7f2eb23b7640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eac0ace00 0x7f2eac15a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.062+0000 7f2eb23b7640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eac15c8e0 0x7f2eac0ac8c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.062+0000 7f2eb23b7640 1 --2- 172.21.15.16:0/1146496191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2eac071a00 0x7f2eac0b3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.062+0000 7f2eb23b7640 1 -- 172.21.15.16:0/1146496191 >> 172.21.15.16:0/1146496191 conn(0x7f2eac0b2000 msgr2=0x7f2eac15b610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:50.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.062+0000 7f2eb23b7640 1 -- 172.21.15.16:0/1146496191 shutdown_connections 2024-04-24T23:49:50.063 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.062+0000 7f2eb23b7640 1 -- 172.21.15.16:0/1146496191 wait complete. 2024-04-24T23:49:50.073 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:49:50.073 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:49:50.090 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:50.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.231+0000 7f1e48283640 1 Processor -- start 2024-04-24T23:49:50.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.231+0000 7f1e48283640 1 -- start start 2024-04-24T23:49:50.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.232+0000 7f1e48283640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1e401457b0 0x7f1e400ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.232+0000 7f1e48283640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e4015f990 0x7f1e400cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.232+0000 7f1e48283640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e4015fe90 con 0x7f1e40144b20 2024-04-24T23:49:50.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.232+0000 7f1e48283640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f1e4006ea50 con 0x7f1e401457b0 2024-04-24T23:49:50.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.232+0000 7f1e48283640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1e40160170 con 0x7f1e4015f990 2024-04-24T23:49:50.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.232+0000 7f1e4601f640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1e40144b20 0x7f1e40144f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42288/0 (socket says 172.21.15.16:42288) 2024-04-24T23:49:50.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.232+0000 7f1e4601f640 1 -- 172.21.15.16:0/2883128331 learned_addr learned my addr 172.21.15.16:0/2883128331 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:50.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.232+0000 7f1e4581e640 1 --2- 172.21.15.16:0/2883128331 >> v2:172.21.15.156:3300/0 conn(0x7f1e401457b0 0x7f1e400ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.232+0000 7f1e46820640 1 --2- 172.21.15.16:0/2883128331 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e4015f990 0x7f1e400cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.233+0000 7f1e4501d640 1 -- 172.21.15.16:0/2883128331 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1082253295 0 0) 0x7f1e4015fe90 con 0x7f1e40144b20 2024-04-24T23:49:50.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.233+0000 7f1e4501d640 1 -- 172.21.15.16:0/2883128331 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e28003520 con 0x7f1e40144b20 2024-04-24T23:49:50.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.233+0000 7f1e4581e640 1 -- 172.21.15.16:0/2883128331 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e4015f990 msgr2=0x7f1e400cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:50.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.233+0000 7f1e4581e640 1 --2- 172.21.15.16:0/2883128331 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e4015f990 0x7f1e400cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.233+0000 7f1e4501d640 1 -- 172.21.15.16:0/2883128331 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2516336160 0 0) 0x7f1e28003520 con 0x7f1e40144b20 2024-04-24T23:49:50.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.233+0000 7f1e4581e640 1 -- 172.21.15.16:0/2883128331 >> v1:172.21.15.16:6789/0 conn(0x7f1e40144b20 legacy=0x7f1e40144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.233+0000 7f1e4581e640 1 -- 172.21.15.16:0/2883128331 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e40160450 con 0x7f1e401457b0 2024-04-24T23:49:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.234+0000 7f1e4581e640 1 --2- 172.21.15.16:0/2883128331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e401457b0 0x7f1e400ce680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1e340097a0 tx=0x7f1e3402a080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb8d887623c0e354 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:50.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.234+0000 7f1e4501d640 1 -- 172.21.15.16:0/2883128331 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e34037070 con 0x7f1e401457b0 2024-04-24T23:49:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.234+0000 7f1e4501d640 1 -- 172.21.15.16:0/2883128331 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e3402ab10 con 0x7f1e401457b0 2024-04-24T23:49:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.234+0000 7f1e4501d640 1 -- 172.21.15.16:0/2883128331 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e38001670 con 0x7f1e401457b0 2024-04-24T23:49:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.234+0000 7f1e48283640 1 -- 172.21.15.16:0/2883128331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e401457b0 msgr2=0x7f1e400ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.234+0000 7f1e48283640 1 --2- 172.21.15.16:0/2883128331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e401457b0 0x7f1e400ce680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1e340097a0 tx=0x7f1e3402a080 comp rx=0 tx=0).stop 2024-04-24T23:49:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.234+0000 7f1e48283640 1 -- 172.21.15.16:0/2883128331 shutdown_connections 2024-04-24T23:49:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.234+0000 7f1e48283640 1 --2- 172.21.15.16:0/2883128331 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e4015f990 0x7f1e400cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.234+0000 7f1e48283640 1 --2- 172.21.15.16:0/2883128331 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e401457b0 0x7f1e400ce680 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.235+0000 7f1e48283640 1 -- 172.21.15.16:0/2883128331 >> 172.21.15.16:0/2883128331 conn(0x7f1e400b2000 msgr2=0x7f1e400cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.235+0000 7f1e48283640 1 -- 172.21.15.16:0/2883128331 shutdown_connections 2024-04-24T23:49:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.235+0000 7f1e48283640 1 -- 172.21.15.16:0/2883128331 wait complete. 2024-04-24T23:49:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.236+0000 7f1e48283640 1 Processor -- start 2024-04-24T23:49:50.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.236+0000 7f1e48283640 1 -- start start 2024-04-24T23:49:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.236+0000 7f1e48283640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e40144b20 0x7f1e401409a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.236+0000 7f1e48283640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e401457b0 0x7f1e4013b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.236+0000 7f1e48283640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e4015f990 0x7f1e4015e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.236+0000 7f1e48283640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1e400b3390 con 0x7f1e401457b0 2024-04-24T23:49:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.236+0000 7f1e48283640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1e4006d4b0 con 0x7f1e4015f990 2024-04-24T23:49:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.236+0000 7f1e48283640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1e4006da40 con 0x7f1e40144b20 2024-04-24T23:49:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e4581e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e401457b0 0x7f1e4013b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e4581e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e401457b0 0x7f1e4013b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40674/0 (socket says 172.21.15.16:40674) 2024-04-24T23:49:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e4581e640 1 -- 172.21.15.16:0/324675039 learned_addr learned my addr 172.21.15.16:0/324675039 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e4601f640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e40144b20 0x7f1e401409a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e46820640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e4015f990 0x7f1e4015e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e4581e640 1 -- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e40144b20 msgr2=0x7f1e401409a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e4581e640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e40144b20 0x7f1e401409a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e4581e640 1 -- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e4015f990 msgr2=0x7f1e4015e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e4581e640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e4015f990 0x7f1e4015e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e4581e640 1 -- 172.21.15.16:0/324675039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e401f5ba0 con 0x7f1e401457b0 2024-04-24T23:49:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e46820640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e4015f990 0x7f1e4015e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e4581e640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e401457b0 0x7f1e4013b320 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1e34035040 tx=0x7f1e340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e3effd640 1 -- 172.21.15.16:0/324675039 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e34037050 con 0x7f1e401457b0 2024-04-24T23:49:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e3effd640 1 -- 172.21.15.16:0/324675039 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e34043070 con 0x7f1e401457b0 2024-04-24T23:49:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e48283640 1 -- 172.21.15.16:0/324675039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e401f5e80 con 0x7f1e401457b0 2024-04-24T23:49:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e3effd640 1 -- 172.21.15.16:0/324675039 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e34006e10 con 0x7f1e401457b0 2024-04-24T23:49:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.237+0000 7f1e48283640 1 -- 172.21.15.16:0/324675039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e401f6440 con 0x7f1e401457b0 2024-04-24T23:49:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.238+0000 7f1e48283640 1 -- 172.21.15.16:0/324675039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e400731f0 con 0x7f1e401457b0 2024-04-24T23:49:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.239+0000 7f1e3effd640 1 -- 172.21.15.16:0/324675039 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1e34048460 con 0x7f1e401457b0 2024-04-24T23:49:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.239+0000 7f1e3effd640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1e2c06be30 0x7f1e2c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.239+0000 7f1e3effd640 1 -- 172.21.15.16:0/324675039 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1e340ab980 con 0x7f1e401457b0 2024-04-24T23:49:50.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.239+0000 7f1e4601f640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1e2c06be30 0x7f1e2c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.240+0000 7f1e4601f640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1e2c06be30 0x7f1e2c06e2f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1e38005880 tx=0x7f1e380057f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:50.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.243+0000 7f1e3effd640 1 -- 172.21.15.16:0/324675039 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1e34079f30 con 0x7f1e401457b0 2024-04-24T23:49:50.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.426+0000 7f1e48283640 1 -- 172.21.15.16:0/324675039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1e4004f5e0 con 0x7f1e401457b0 2024-04-24T23:49:50.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.427+0000 7f1e3effd640 1 -- 172.21.15.16:0/324675039 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7f1e34079830 con 0x7f1e401457b0 2024-04-24T23:49:50.428 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:50.428 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:50.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.429+0000 7f1e48283640 1 -- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1e2c06be30 msgr2=0x7f1e2c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.429+0000 7f1e48283640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1e2c06be30 0x7f1e2c06e2f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1e38005880 tx=0x7f1e380057f0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.429+0000 7f1e48283640 1 -- 172.21.15.16:0/324675039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e401457b0 msgr2=0x7f1e4013b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.429+0000 7f1e48283640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e401457b0 0x7f1e4013b320 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1e34035040 tx=0x7f1e340046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.429+0000 7f1e48283640 1 -- 172.21.15.16:0/324675039 shutdown_connections 2024-04-24T23:49:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.429+0000 7f1e48283640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1e2c06be30 0x7f1e2c06e2f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.429+0000 7f1e48283640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e4015f990 0x7f1e4015e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.429+0000 7f1e48283640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e401457b0 0x7f1e4013b320 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.429+0000 7f1e48283640 1 --2- 172.21.15.16:0/324675039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e40144b20 0x7f1e401409a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.429+0000 7f1e48283640 1 -- 172.21.15.16:0/324675039 >> 172.21.15.16:0/324675039 conn(0x7f1e400b2000 msgr2=0x7f1e40070fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.430+0000 7f1e48283640 1 -- 172.21.15.16:0/324675039 shutdown_connections 2024-04-24T23:49:50.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.430+0000 7f1e48283640 1 -- 172.21.15.16:0/324675039 wait complete. 2024-04-24T23:49:50.440 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:50.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:50.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.546+0000 7faa424f5640 1 Processor -- start 2024-04-24T23:49:50.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.546+0000 7faa424f5640 1 -- start start 2024-04-24T23:49:50.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.547+0000 7faa424f5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faa3c06c0d0 0x7faa3c0b7760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.547+0000 7faa424f5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa3c0d24d0 con 0x7faa3c15dfd0 2024-04-24T23:49:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.547+0000 7faa424f5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa3c0d27b0 con 0x7faa3c06bc40 2024-04-24T23:49:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.547+0000 7faa424f5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7faa3c06ba00 con 0x7faa3c06c0d0 2024-04-24T23:49:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.547+0000 7faa3bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faa3c15dfd0 0x7faa3c06b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42304/0 (socket says 172.21.15.16:42304) 2024-04-24T23:49:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.547+0000 7faa3bfff640 1 -- 172.21.15.16:0/674412666 learned_addr learned my addr 172.21.15.16:0/674412666 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 -- 172.21.15.16:0/674412666 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3594788454 0 0) 0x7faa3c0d24d0 con 0x7faa3c15dfd0 2024-04-24T23:49:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 -- 172.21.15.16:0/674412666 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faa24003520 con 0x7faa3c15dfd0 2024-04-24T23:49:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa40a92640 1 --2- 172.21.15.16:0/674412666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faa3c06c0d0 0x7faa3c0b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 -- 172.21.15.16:0/674412666 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3872391053 0 0) 0x7faa24003520 con 0x7faa3c15dfd0 2024-04-24T23:49:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 -- 172.21.15.16:0/674412666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faa3c06c0d0 msgr2=0x7faa3c0b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 --2- 172.21.15.16:0/674412666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faa3c06c0d0 0x7faa3c0b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 -- 172.21.15.16:0/674412666 >> v1:172.21.15.156:6789/0 conn(0x7faa3c06bc40 legacy=0x7faa3c0b7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 -- 172.21.15.16:0/674412666 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa3c0d2a90 con 0x7faa3c15dfd0 2024-04-24T23:49:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 -- 172.21.15.16:0/674412666 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7faa2c001670 con 0x7faa3c15dfd0 2024-04-24T23:49:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 -- 172.21.15.16:0/674412666 >> v1:172.21.15.16:6789/0 conn(0x7faa3c15dfd0 legacy=0x7faa3c06b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 --2- 172.21.15.16:0/674412666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa240039e0 0x7faa24003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 --2- 172.21.15.16:0/674412666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faa3c06c0d0 0x7faa24004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 --2- 172.21.15.16:0/674412666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa24004d40 0x7faa240051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 -- 172.21.15.16:0/674412666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c06ba00 con 0x7faa24004d40 2024-04-24T23:49:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 -- 172.21.15.16:0/674412666 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faa2400baf0 con 0x7faa240039e0 2024-04-24T23:49:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3affd640 1 -- 172.21.15.16:0/674412666 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7faa2400bcf0 con 0x7faa3c06c0d0 2024-04-24T23:49:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa40a92640 1 --2- 172.21.15.16:0/674412666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa24004d40 0x7faa240051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3bfff640 1 --2- 172.21.15.16:0/674412666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa240039e0 0x7faa24003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa3b7fe640 1 --2- 172.21.15.16:0/674412666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faa3c06c0d0 0x7faa24004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa40a92640 1 -- 172.21.15.16:0/674412666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faa3c06c0d0 msgr2=0x7faa24004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa40a92640 1 --2- 172.21.15.16:0/674412666 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faa3c06c0d0 0x7faa24004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.551 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa40a92640 1 -- 172.21.15.16:0/674412666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa240039e0 msgr2=0x7faa24003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa40a92640 1 --2- 172.21.15.16:0/674412666 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa240039e0 0x7faa24003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.548+0000 7faa40a92640 1 -- 172.21.15.16:0/674412666 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa24003520 con 0x7faa24004d40 2024-04-24T23:49:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.549+0000 7faa40a92640 1 --2- 172.21.15.16:0/674412666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa24004d40 0x7faa240051c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faa34009fc0 tx=0x7faa24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a5d543b61152865 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.549+0000 7faa3affd640 1 -- 172.21.15.16:0/674412666 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34009c10 con 0x7faa24004d40 2024-04-24T23:49:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.549+0000 7faa3affd640 1 -- 172.21.15.16:0/674412666 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa34036070 con 0x7faa24004d40 2024-04-24T23:49:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.549+0000 7faa3affd640 1 -- 172.21.15.16:0/674412666 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34031880 con 0x7faa24004d40 2024-04-24T23:49:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.550+0000 7faa424f5640 1 -- 172.21.15.16:0/674412666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa24004d40 msgr2=0x7faa240051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.550+0000 7faa424f5640 1 --2- 172.21.15.16:0/674412666 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa24004d40 0x7faa240051c0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faa34009fc0 tx=0x7faa24005830 comp rx=0 tx=0).stop 2024-04-24T23:49:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.550+0000 7faa40a92640 1 -- 172.21.15.16:0/674412666 reap_dead start 2024-04-24T23:49:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.550+0000 7faa424f5640 1 -- 172.21.15.16:0/674412666 shutdown_connections 2024-04-24T23:49:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.550+0000 7faa424f5640 1 -- 172.21.15.16:0/674412666 >> 172.21.15.16:0/674412666 conn(0x7faa3c0d3dc0 msgr2=0x7faa3c0d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.550+0000 7faa424f5640 1 -- 172.21.15.16:0/674412666 shutdown_connections 2024-04-24T23:49:50.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.550+0000 7faa424f5640 1 -- 172.21.15.16:0/674412666 wait complete. 2024-04-24T23:49:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.551+0000 7faa424f5640 1 Processor -- start 2024-04-24T23:49:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.551+0000 7faa424f5640 1 -- start start 2024-04-24T23:49:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa424f5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa3c15aa10 0x7faa3c15b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa424f5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faa3c15adf0 0x7faa3c15b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa424f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c0caf00 0x7faa3c0cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa424f5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c06c520 con 0x7faa3c0caf00 2024-04-24T23:49:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa424f5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7faa3c0d3050 con 0x7faa3c15aa10 2024-04-24T23:49:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa424f5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7faa3c0cead0 con 0x7faa3c15adf0 2024-04-24T23:49:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa40a92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c0caf00 0x7faa3c0cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa40a92640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c0caf00 0x7faa3c0cb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40692/0 (socket says 172.21.15.16:40692) 2024-04-24T23:49:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa40a92640 1 -- 172.21.15.16:0/2688025468 learned_addr learned my addr 172.21.15.16:0/2688025468 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa3bfff640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa3c15aa10 0x7faa3c15b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa3b7fe640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faa3c15adf0 0x7faa3c15b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa40a92640 1 -- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faa3c15adf0 msgr2=0x7faa3c15b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa40a92640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faa3c15adf0 0x7faa3c15b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa40a92640 1 -- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa3c15aa10 msgr2=0x7faa3c15b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa40a92640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa3c15aa10 0x7faa3c15b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.552+0000 7faa40a92640 1 -- 172.21.15.16:0/2688025468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa3c0cbb10 con 0x7faa3c0caf00 2024-04-24T23:49:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.553+0000 7faa3bfff640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa3c15aa10 0x7faa3c15b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.553+0000 7faa40a92640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c0caf00 0x7faa3c0cb380 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7faa3400e230 tx=0x7faa34009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.553+0000 7faa38ff9640 1 -- 172.21.15.16:0/2688025468 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34009c10 con 0x7faa3c0caf00 2024-04-24T23:49:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.553+0000 7faa38ff9640 1 -- 172.21.15.16:0/2688025468 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa34031d30 con 0x7faa3c0caf00 2024-04-24T23:49:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.553+0000 7faa424f5640 1 -- 172.21.15.16:0/2688025468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa3c0cbdf0 con 0x7faa3c0caf00 2024-04-24T23:49:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.553+0000 7faa38ff9640 1 -- 172.21.15.16:0/2688025468 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34003bd0 con 0x7faa3c0caf00 2024-04-24T23:49:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.553+0000 7faa424f5640 1 -- 172.21.15.16:0/2688025468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa3c0cc3b0 con 0x7faa3c0caf00 2024-04-24T23:49:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.554+0000 7faa424f5640 1 -- 172.21.15.16:0/2688025468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa3c068f90 con 0x7faa3c0caf00 2024-04-24T23:49:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.554+0000 7faa38ff9640 1 -- 172.21.15.16:0/2688025468 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7faa34005d10 con 0x7faa3c0caf00 2024-04-24T23:49:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.555+0000 7faa38ff9640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7faa2806be60 0x7faa2806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.555+0000 7faa38ff9640 1 -- 172.21.15.16:0/2688025468 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7faa340ac5c0 con 0x7faa3c0caf00 2024-04-24T23:49:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.555+0000 7faa3bfff640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7faa2806be60 0x7faa2806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.555+0000 7faa3bfff640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7faa2806be60 0x7faa2806e320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0013b0 tx=0x7faa2c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:50.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.559+0000 7faa38ff9640 1 -- 172.21.15.16:0/2688025468 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7faa3407aca0 con 0x7faa3c0caf00 2024-04-24T23:49:50.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.743+0000 7faa424f5640 1 -- 172.21.15.16:0/2688025468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faa3c0b5640 con 0x7faa3c0caf00 2024-04-24T23:49:50.742 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.743+0000 7faa38ff9640 1 -- 172.21.15.16:0/2688025468 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7faa3407a5a0 con 0x7faa3c0caf00 2024-04-24T23:49:50.743 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.745+0000 7faa424f5640 1 -- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7faa2806be60 msgr2=0x7faa2806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.745+0000 7faa424f5640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7faa2806be60 0x7faa2806e320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7faa2c0013b0 tx=0x7faa2c009040 comp rx=0 tx=0).stop 2024-04-24T23:49:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.745+0000 7faa424f5640 1 -- 172.21.15.16:0/2688025468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c0caf00 msgr2=0x7faa3c0cb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.745+0000 7faa424f5640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c0caf00 0x7faa3c0cb380 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7faa3400e230 tx=0x7faa34009750 comp rx=0 tx=0).stop 2024-04-24T23:49:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.745+0000 7faa424f5640 1 -- 172.21.15.16:0/2688025468 shutdown_connections 2024-04-24T23:49:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.745+0000 7faa424f5640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7faa2806be60 0x7faa2806e320 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.745+0000 7faa424f5640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa3c0caf00 0x7faa3c0cb380 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.745+0000 7faa424f5640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7faa3c15adf0 0x7faa3c15b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.745+0000 7faa424f5640 1 --2- 172.21.15.16:0/2688025468 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7faa3c15aa10 0x7faa3c15b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.745+0000 7faa424f5640 1 -- 172.21.15.16:0/2688025468 >> 172.21.15.16:0/2688025468 conn(0x7faa3c0d3dc0 msgr2=0x7faa3c0b5b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.745+0000 7faa424f5640 1 -- 172.21.15.16:0/2688025468 shutdown_connections 2024-04-24T23:49:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.745+0000 7faa424f5640 1 -- 172.21.15.16:0/2688025468 wait complete. 2024-04-24T23:49:50.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:50.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.859+0000 7fa07c42a640 1 Processor -- start 2024-04-24T23:49:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.859+0000 7fa07c42a640 1 -- start start 2024-04-24T23:49:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.860+0000 7fa07c42a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0741455f0 0x7fa07406b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.860+0000 7fa07c42a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa0740cee70 0x7fa07406ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.860+0000 7fa07c42a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa07415f9c0 con 0x7fa07406b620 2024-04-24T23:49:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.860+0000 7fa07c42a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa0741605d0 con 0x7fa0740cee70 2024-04-24T23:49:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.860+0000 7fa07c42a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fa07415fca0 con 0x7fa0741455f0 2024-04-24T23:49:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.860+0000 7fa07a1c6640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0741455f0 0x7fa07406b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.860+0000 7fa07a9c7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa0740cee70 0x7fa07406ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.860+0000 7fa0799c5640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fa07406b620 0x7fa0740ce760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:60974/0 (socket says 172.21.15.16:60974) 2024-04-24T23:49:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.860+0000 7fa0799c5640 1 -- 172.21.15.16:0/2283452408 learned_addr learned my addr 172.21.15.16:0/2283452408 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.861+0000 7fa07a1c6640 1 -- 172.21.15.16:0/2283452408 >> v2:172.21.15.156:3300/0 conn(0x7fa0740cee70 msgr2=0x7fa07406ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.861+0000 7fa07a1c6640 1 --2- 172.21.15.16:0/2283452408 >> v2:172.21.15.156:3300/0 conn(0x7fa0740cee70 0x7fa07406ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.861+0000 7fa07a1c6640 1 -- 172.21.15.16:0/2283452408 >> v1:172.21.15.156:6789/0 conn(0x7fa07406b620 legacy=0x7fa0740ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.861+0000 7fa07a1c6640 1 -- 172.21.15.16:0/2283452408 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa07415ff80 con 0x7fa0741455f0 2024-04-24T23:49:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.861+0000 7fa07a9c7640 1 --2- 172.21.15.16:0/2283452408 >> v2:172.21.15.156:3300/0 conn(0x7fa0740cee70 0x7fa07406ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:49:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.861+0000 7fa07a1c6640 1 --2- 172.21.15.16:0/2283452408 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0741455f0 0x7fa07406b0e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa0700097a0 tx=0x7fa07002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f17dc1612873eae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.861+0000 7fa0791c4640 1 -- 172.21.15.16:0/2283452408 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa070037070 con 0x7fa0741455f0 2024-04-24T23:49:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.862+0000 7fa0791c4640 1 -- 172.21.15.16:0/2283452408 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa07002ae50 con 0x7fa0741455f0 2024-04-24T23:49:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.862+0000 7fa0791c4640 1 -- 172.21.15.16:0/2283452408 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa070032660 con 0x7fa0741455f0 2024-04-24T23:49:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.862+0000 7fa07c42a640 1 -- 172.21.15.16:0/2283452408 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0741455f0 msgr2=0x7fa07406b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.862+0000 7fa07c42a640 1 --2- 172.21.15.16:0/2283452408 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0741455f0 0x7fa07406b0e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa0700097a0 tx=0x7fa07002a100 comp rx=0 tx=0).stop 2024-04-24T23:49:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.862+0000 7fa07c42a640 1 -- 172.21.15.16:0/2283452408 shutdown_connections 2024-04-24T23:49:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.862+0000 7fa07c42a640 1 --2- 172.21.15.16:0/2283452408 >> v2:172.21.15.156:3300/0 conn(0x7fa0740cee70 0x7fa07406ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.862+0000 7fa07c42a640 1 --2- 172.21.15.16:0/2283452408 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0741455f0 0x7fa07406b0e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.862+0000 7fa07c42a640 1 -- 172.21.15.16:0/2283452408 >> 172.21.15.16:0/2283452408 conn(0x7fa0740cfc50 msgr2=0x7fa0740d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.862+0000 7fa07c42a640 1 -- 172.21.15.16:0/2283452408 shutdown_connections 2024-04-24T23:49:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.862+0000 7fa07c42a640 1 -- 172.21.15.16:0/2283452408 wait complete. 2024-04-24T23:49:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.863+0000 7fa07c42a640 1 Processor -- start 2024-04-24T23:49:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.863+0000 7fa07c42a640 1 -- start start 2024-04-24T23:49:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa07c42a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa07406b620 0x7fa074140120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa07c42a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0740cee70 0x7fa07414e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa07c42a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0741455f0 0x7fa0741f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa07c42a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa07415d0a0 con 0x7fa0741455f0 2024-04-24T23:49:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa07c42a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa0741459d0 con 0x7fa07406b620 2024-04-24T23:49:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa07c42a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fa07415d220 con 0x7fa0740cee70 2024-04-24T23:49:50.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa07a9c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0741455f0 0x7fa0741f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa07a9c7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0741455f0 0x7fa0741f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40700/0 (socket says 172.21.15.16:40700) 2024-04-24T23:49:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa07a9c7640 1 -- 172.21.15.16:0/4048844108 learned_addr learned my addr 172.21.15.16:0/4048844108 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa07a1c6640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa07406b620 0x7fa074140120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa0799c5640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0740cee70 0x7fa07414e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa07a9c7640 1 -- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0740cee70 msgr2=0x7fa07414e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa07a9c7640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0740cee70 0x7fa07414e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa07a9c7640 1 -- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa07406b620 msgr2=0x7fa074140120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.864+0000 7fa07a9c7640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa07406b620 0x7fa074140120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.865+0000 7fa07a9c7640 1 -- 172.21.15.16:0/4048844108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0740cd4d0 con 0x7fa0741455f0 2024-04-24T23:49:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.865+0000 7fa07a1c6640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa07406b620 0x7fa074140120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:50.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.865+0000 7fa07a9c7640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0741455f0 0x7fa0741f6bd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa06c009f00 tx=0x7fa06c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:50.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.865+0000 7fa06affd640 1 -- 172.21.15.16:0/4048844108 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa06c018030 con 0x7fa0741455f0 2024-04-24T23:49:50.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.865+0000 7fa06affd640 1 -- 172.21.15.16:0/4048844108 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa06c00ce50 con 0x7fa0741455f0 2024-04-24T23:49:50.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.865+0000 7fa07c42a640 1 -- 172.21.15.16:0/4048844108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0740cd7b0 con 0x7fa0741455f0 2024-04-24T23:49:50.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.865+0000 7fa06affd640 1 -- 172.21.15.16:0/4048844108 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa06c003bb0 con 0x7fa0741455f0 2024-04-24T23:49:50.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.865+0000 7fa07c42a640 1 -- 172.21.15.16:0/4048844108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0740cdd70 con 0x7fa0741455f0 2024-04-24T23:49:50.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.866+0000 7fa07c42a640 1 -- 172.21.15.16:0/4048844108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa074068f90 con 0x7fa0741455f0 2024-04-24T23:49:50.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.866+0000 7fa06affd640 1 -- 172.21.15.16:0/4048844108 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa06c01b070 con 0x7fa0741455f0 2024-04-24T23:49:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.867+0000 7fa06affd640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa05806be30 0x7fa05806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.867+0000 7fa06affd640 1 -- 172.21.15.16:0/4048844108 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa06c08bdf0 con 0x7fa0741455f0 2024-04-24T23:49:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.867+0000 7fa07a1c6640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa05806be30 0x7fa05806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.868+0000 7fa07a1c6640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa05806be30 0x7fa05806e2f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa070034000 tx=0x7fa07002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:50.872+0000 7fa06affd640 1 -- 172.21.15.16:0/4048844108 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa06c010040 con 0x7fa0741455f0 2024-04-24T23:49:51.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.054+0000 7fa07c42a640 1 -- 172.21.15.16:0/4048844108 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa07415bdd0 con 0x7fa0741455f0 2024-04-24T23:49:51.054 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.055+0000 7fa06affd640 1 -- 172.21.15.16:0/4048844108 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fa06c012340 con 0x7fa0741455f0 2024-04-24T23:49:51.055 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.057+0000 7fa07c42a640 1 -- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa05806be30 msgr2=0x7fa05806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.057+0000 7fa07c42a640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa05806be30 0x7fa05806e2f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fa070034000 tx=0x7fa07002e000 comp rx=0 tx=0).stop 2024-04-24T23:49:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.057+0000 7fa07c42a640 1 -- 172.21.15.16:0/4048844108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0741455f0 msgr2=0x7fa0741f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.057+0000 7fa07c42a640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0741455f0 0x7fa0741f6bd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa06c009f00 tx=0x7fa06c00c490 comp rx=0 tx=0).stop 2024-04-24T23:49:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.057+0000 7fa07c42a640 1 -- 172.21.15.16:0/4048844108 shutdown_connections 2024-04-24T23:49:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.057+0000 7fa07c42a640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa05806be30 0x7fa05806e2f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.057+0000 7fa07c42a640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0741455f0 0x7fa0741f6bd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.057+0000 7fa07c42a640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0740cee70 0x7fa07414e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.057+0000 7fa07c42a640 1 --2- 172.21.15.16:0/4048844108 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa07406b620 0x7fa074140120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.057+0000 7fa07c42a640 1 -- 172.21.15.16:0/4048844108 >> 172.21.15.16:0/4048844108 conn(0x7fa0740cfc50 msgr2=0x7fa0740b53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.057+0000 7fa07c42a640 1 -- 172.21.15.16:0/4048844108 shutdown_connections 2024-04-24T23:49:51.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.057+0000 7fa07c42a640 1 -- 172.21.15.16:0/4048844108 wait complete. 2024-04-24T23:49:51.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:51.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.172+0000 7fb94e464640 1 Processor -- start 2024-04-24T23:49:51.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.172+0000 7fb94e464640 1 -- start start 2024-04-24T23:49:51.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.172+0000 7fb94e464640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb94806bc40 0x7fb94806c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.172+0000 7fb94e464640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9480b7050 0x7fb9480b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.172+0000 7fb94e464640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9480d24d0 con 0x7fb94815dfd0 2024-04-24T23:49:51.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.172+0000 7fb94e464640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb94806ba00 con 0x7fb94806bc40 2024-04-24T23:49:51.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.172+0000 7fb94e464640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb9480d27b0 con 0x7fb9480b7050 2024-04-24T23:49:51.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.173+0000 7fb9477fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb94806bc40 0x7fb94806c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.173+0000 7fb947fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb94815dfd0 0x7fb94806b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42314/0 (socket says 172.21.15.16:42314) 2024-04-24T23:49:51.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.173+0000 7fb947fff640 1 -- 172.21.15.16:0/2163829006 learned_addr learned my addr 172.21.15.16:0/2163829006 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:51.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.173+0000 7fb94ca01640 1 --2- 172.21.15.16:0/2163829006 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9480b7050 0x7fb9480b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.173+0000 7fb9477fe640 1 -- 172.21.15.16:0/2163829006 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9480b7050 msgr2=0x7fb9480b74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:51.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.173+0000 7fb9477fe640 1 --2- 172.21.15.16:0/2163829006 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9480b7050 0x7fb9480b74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.173+0000 7fb9477fe640 1 -- 172.21.15.16:0/2163829006 >> v1:172.21.15.16:6789/0 conn(0x7fb94815dfd0 legacy=0x7fb94806b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.173+0000 7fb946ffd640 1 -- 172.21.15.16:0/2163829006 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3834074807 0 0) 0x7fb9480d24d0 con 0x7fb94815dfd0 2024-04-24T23:49:51.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.173+0000 7fb9477fe640 1 -- 172.21.15.16:0/2163829006 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9480d2a90 con 0x7fb94806bc40 2024-04-24T23:49:51.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.174+0000 7fb9477fe640 1 --2- 172.21.15.16:0/2163829006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb94806bc40 0x7fb94806c0d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0097a0 tx=0x7fb93c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7439c99f2d7c5e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:51.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.174+0000 7fb946ffd640 1 -- 172.21.15.16:0/2163829006 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c037070 con 0x7fb94806bc40 2024-04-24T23:49:51.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.174+0000 7fb946ffd640 1 -- 172.21.15.16:0/2163829006 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb93c02ae50 con 0x7fb94806bc40 2024-04-24T23:49:51.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.174+0000 7fb946ffd640 1 -- 172.21.15.16:0/2163829006 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb93c032660 con 0x7fb94806bc40 2024-04-24T23:49:51.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.174+0000 7fb94e464640 1 -- 172.21.15.16:0/2163829006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb94806bc40 msgr2=0x7fb94806c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.174+0000 7fb94e464640 1 --2- 172.21.15.16:0/2163829006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb94806bc40 0x7fb94806c0d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0097a0 tx=0x7fb93c02a100 comp rx=0 tx=0).stop 2024-04-24T23:49:51.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.174+0000 7fb94e464640 1 -- 172.21.15.16:0/2163829006 shutdown_connections 2024-04-24T23:49:51.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.174+0000 7fb94e464640 1 --2- 172.21.15.16:0/2163829006 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9480b7050 0x7fb9480b74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.174+0000 7fb94e464640 1 --2- 172.21.15.16:0/2163829006 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb94806bc40 0x7fb94806c0d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.174+0000 7fb94e464640 1 -- 172.21.15.16:0/2163829006 >> 172.21.15.16:0/2163829006 conn(0x7fb9480d3dc0 msgr2=0x7fb9480d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:51.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.174+0000 7fb94e464640 1 -- 172.21.15.16:0/2163829006 shutdown_connections 2024-04-24T23:49:51.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.174+0000 7fb94e464640 1 -- 172.21.15.16:0/2163829006 wait complete. 2024-04-24T23:49:51.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.176+0000 7fb94e464640 1 Processor -- start 2024-04-24T23:49:51.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.176+0000 7fb94e464640 1 -- start start 2024-04-24T23:49:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.176+0000 7fb94e464640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb94806bc40 0x7fb9480c9940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.176+0000 7fb94e464640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9480b7050 0x7fb9481e7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.176+0000 7fb94e464640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb94815dfd0 0x7fb9481e6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.176+0000 7fb94e464640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb9480cead0 con 0x7fb94815dfd0 2024-04-24T23:49:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.176+0000 7fb94e464640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb9480d3050 con 0x7fb94806bc40 2024-04-24T23:49:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.176+0000 7fb94e464640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb9480b79e0 con 0x7fb9480b7050 2024-04-24T23:49:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb94ca01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb94815dfd0 0x7fb9481e6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb94ca01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb94815dfd0 0x7fb9481e6470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40712/0 (socket says 172.21.15.16:40712) 2024-04-24T23:49:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb94ca01640 1 -- 172.21.15.16:0/1974629798 learned_addr learned my addr 172.21.15.16:0/1974629798 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb947fff640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb94806bc40 0x7fb9480c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb9477fe640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9480b7050 0x7fb9481e7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb94ca01640 1 -- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9480b7050 msgr2=0x7fb9481e7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb94ca01640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9480b7050 0x7fb9481e7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb94ca01640 1 -- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb94806bc40 msgr2=0x7fb9480c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb94ca01640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb94806bc40 0x7fb9480c9940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb94ca01640 1 -- 172.21.15.16:0/1974629798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9481e6f30 con 0x7fb94815dfd0 2024-04-24T23:49:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb9477fe640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9480b7050 0x7fb9481e7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb94ca01640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb94815dfd0 0x7fb9481e6470 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb940007e40 tx=0x7fb94000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb944ff9640 1 -- 172.21.15.16:0/1974629798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb940014620 con 0x7fb94815dfd0 2024-04-24T23:49:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb944ff9640 1 -- 172.21.15.16:0/1974629798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb940019070 con 0x7fb94815dfd0 2024-04-24T23:49:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb944ff9640 1 -- 172.21.15.16:0/1974629798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb940014a20 con 0x7fb94815dfd0 2024-04-24T23:49:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb94e464640 1 -- 172.21.15.16:0/1974629798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9481e7210 con 0x7fb94815dfd0 2024-04-24T23:49:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.177+0000 7fb94e464640 1 -- 172.21.15.16:0/1974629798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9481e77d0 con 0x7fb94815dfd0 2024-04-24T23:49:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.178+0000 7fb94e464640 1 -- 172.21.15.16:0/1974629798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9480ca440 con 0x7fb94815dfd0 2024-04-24T23:49:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.179+0000 7fb944ff9640 1 -- 172.21.15.16:0/1974629798 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb940014bc0 con 0x7fb94815dfd0 2024-04-24T23:49:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.179+0000 7fb944ff9640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb93406be30 0x7fb93406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.179+0000 7fb944ff9640 1 -- 172.21.15.16:0/1974629798 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb94008bae0 con 0x7fb94815dfd0 2024-04-24T23:49:51.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.179+0000 7fb947fff640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb93406be30 0x7fb93406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.180+0000 7fb947fff640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb93406be30 0x7fb93406e2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb938000f60 tx=0x7fb938007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:51.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.183+0000 7fb944ff9640 1 -- 172.21.15.16:0/1974629798 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb9480ca440 con 0x7fb94815dfd0 2024-04-24T23:49:51.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.368+0000 7fb94e464640 1 -- 172.21.15.16:0/1974629798 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb948068f90 con 0x7fb94815dfd0 2024-04-24T23:49:51.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.368+0000 7fb944ff9640 1 -- 172.21.15.16:0/1974629798 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fb94005a120 con 0x7fb94815dfd0 2024-04-24T23:49:51.368 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:51.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.370+0000 7fb94e464640 1 -- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb93406be30 msgr2=0x7fb93406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.370+0000 7fb94e464640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb93406be30 0x7fb93406e2f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb938000f60 tx=0x7fb938007c00 comp rx=0 tx=0).stop 2024-04-24T23:49:51.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.370+0000 7fb94e464640 1 -- 172.21.15.16:0/1974629798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb94815dfd0 msgr2=0x7fb9481e6470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.370+0000 7fb94e464640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb94815dfd0 0x7fb9481e6470 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb940007e40 tx=0x7fb94000e4b0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.370+0000 7fb94e464640 1 -- 172.21.15.16:0/1974629798 shutdown_connections 2024-04-24T23:49:51.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.370+0000 7fb94e464640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb93406be30 0x7fb93406e2f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.370+0000 7fb94e464640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb94815dfd0 0x7fb9481e6470 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.370+0000 7fb94e464640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb9480b7050 0x7fb9481e7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.370+0000 7fb94e464640 1 --2- 172.21.15.16:0/1974629798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb94806bc40 0x7fb9480c9940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.370+0000 7fb94e464640 1 -- 172.21.15.16:0/1974629798 >> 172.21.15.16:0/1974629798 conn(0x7fb9480d3dc0 msgr2=0x7fb9481603c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:51.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.370+0000 7fb94e464640 1 -- 172.21.15.16:0/1974629798 shutdown_connections 2024-04-24T23:49:51.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.370+0000 7fb94e464640 1 -- 172.21.15.16:0/1974629798 wait complete. 2024-04-24T23:49:51.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.485+0000 7f8a279a2640 1 Processor -- start 2024-04-24T23:49:51.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.485+0000 7f8a279a2640 1 -- start start 2024-04-24T23:49:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.486+0000 7f8a279a2640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8a200af980 0x7f8a2006deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.486+0000 7f8a279a2640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8a2006d360 0x7f8a20071e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.486+0000 7f8a279a2640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a2006d8d0 con 0x7f8a2015dff0 2024-04-24T23:49:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.486+0000 7f8a279a2640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8a200732d0 con 0x7f8a2006d360 2024-04-24T23:49:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.486+0000 7f8a279a2640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8a2006dbb0 con 0x7f8a200af980 2024-04-24T23:49:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.486+0000 7f8a25f3f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8a2006d360 0x7f8a20071e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.486+0000 7f8a25f3f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8a2006d360 0x7f8a20071e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40724/0 (socket says 172.21.15.16:40724) 2024-04-24T23:49:51.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.486+0000 7f8a25f3f640 1 -- 172.21.15.16:0/3813837461 learned_addr learned my addr 172.21.15.16:0/3813837461 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.486+0000 7f8a24f3d640 1 --2- 172.21.15.16:0/3813837461 >> v2:172.21.15.156:3300/0 conn(0x7f8a200af980 0x7f8a2006deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.486+0000 7f8a25f3f640 1 -- 172.21.15.16:0/3813837461 >> v2:172.21.15.156:3300/0 conn(0x7f8a200af980 msgr2=0x7f8a2006deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.486+0000 7f8a25f3f640 1 --2- 172.21.15.16:0/3813837461 >> v2:172.21.15.156:3300/0 conn(0x7f8a200af980 0x7f8a2006deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.486+0000 7f8a25f3f640 1 -- 172.21.15.16:0/3813837461 >> v1:172.21.15.156:6789/0 conn(0x7f8a2015dff0 legacy=0x7f8a200af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:51.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.486+0000 7f8a25f3f640 1 -- 172.21.15.16:0/3813837461 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a20071330 con 0x7f8a2006d360 2024-04-24T23:49:51.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.487+0000 7f8a25f3f640 1 --2- 172.21.15.16:0/3813837461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a2006d360 0x7f8a20071e10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8a20068e40 tx=0x7f8a1802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df5e4045cc493f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.487+0000 7f8a17fff640 1 -- 172.21.15.16:0/3813837461 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a18036070 con 0x7f8a2006d360 2024-04-24T23:49:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.487+0000 7f8a17fff640 1 -- 172.21.15.16:0/3813837461 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a18031600 con 0x7f8a2006d360 2024-04-24T23:49:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.487+0000 7f8a17fff640 1 -- 172.21.15.16:0/3813837461 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a180319d0 con 0x7f8a2006d360 2024-04-24T23:49:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.488+0000 7f8a279a2640 1 -- 172.21.15.16:0/3813837461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a2006d360 msgr2=0x7f8a20071e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.488+0000 7f8a279a2640 1 --2- 172.21.15.16:0/3813837461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a2006d360 0x7f8a20071e10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8a20068e40 tx=0x7f8a1802a6f0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.488+0000 7f8a279a2640 1 -- 172.21.15.16:0/3813837461 shutdown_connections 2024-04-24T23:49:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.488+0000 7f8a279a2640 1 --2- 172.21.15.16:0/3813837461 >> v2:172.21.15.156:3300/0 conn(0x7f8a200af980 0x7f8a2006deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.488+0000 7f8a279a2640 1 --2- 172.21.15.16:0/3813837461 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a2006d360 0x7f8a20071e10 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.488+0000 7f8a279a2640 1 -- 172.21.15.16:0/3813837461 >> 172.21.15.16:0/3813837461 conn(0x7f8a200affc0 msgr2=0x7f8a200b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.488+0000 7f8a279a2640 1 -- 172.21.15.16:0/3813837461 shutdown_connections 2024-04-24T23:49:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.488+0000 7f8a279a2640 1 -- 172.21.15.16:0/3813837461 wait complete. 2024-04-24T23:49:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.489+0000 7f8a279a2640 1 Processor -- start 2024-04-24T23:49:51.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.489+0000 7f8a279a2640 1 -- start start 2024-04-24T23:49:51.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a279a2640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a2006d360 0x7f8a20158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a279a2640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a200af980 0x7f8a20148b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a279a2640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a2015dff0 0x7f8a20160a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a279a2640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a200ccca0 con 0x7f8a2006d360 2024-04-24T23:49:51.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a279a2640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8a200718f0 con 0x7f8a2015dff0 2024-04-24T23:49:51.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a279a2640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8a200cce20 con 0x7f8a200af980 2024-04-24T23:49:51.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a2573e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a2006d360 0x7f8a20158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a2573e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a2006d360 0x7f8a20158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40738/0 (socket says 172.21.15.16:40738) 2024-04-24T23:49:51.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a2573e640 1 -- 172.21.15.16:0/1943349710 learned_addr learned my addr 172.21.15.16:0/1943349710 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:51.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a24f3d640 1 --2- 172.21.15.16:0/1943349710 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a200af980 0x7f8a20148b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a25f3f640 1 --2- 172.21.15.16:0/1943349710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a2015dff0 0x7f8a20160a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a2573e640 1 -- 172.21.15.16:0/1943349710 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a200af980 msgr2=0x7f8a20148b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a2573e640 1 --2- 172.21.15.16:0/1943349710 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a200af980 0x7f8a20148b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a2573e640 1 -- 172.21.15.16:0/1943349710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a2015dff0 msgr2=0x7f8a20160a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a2573e640 1 --2- 172.21.15.16:0/1943349710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a2015dff0 0x7f8a20160a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.490+0000 7f8a2573e640 1 -- 172.21.15.16:0/1943349710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a201490c0 con 0x7f8a2006d360 2024-04-24T23:49:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.491+0000 7f8a2573e640 1 --2- 172.21.15.16:0/1943349710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a2006d360 0x7f8a20158350 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00b3f0 tx=0x7f8a1c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.491+0000 7f8a15ffb640 1 -- 172.21.15.16:0/1943349710 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c015020 con 0x7f8a2006d360 2024-04-24T23:49:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.491+0000 7f8a15ffb640 1 -- 172.21.15.16:0/1943349710 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a1c011830 con 0x7f8a2006d360 2024-04-24T23:49:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.491+0000 7f8a279a2640 1 -- 172.21.15.16:0/1943349710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a2015f710 con 0x7f8a2006d360 2024-04-24T23:49:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.491+0000 7f8a15ffb640 1 -- 172.21.15.16:0/1943349710 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c011c20 con 0x7f8a2006d360 2024-04-24T23:49:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.491+0000 7f8a279a2640 1 -- 172.21.15.16:0/1943349710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a2015fcd0 con 0x7f8a2006d360 2024-04-24T23:49:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.492+0000 7f8a279a2640 1 -- 172.21.15.16:0/1943349710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a20068f90 con 0x7f8a2006d360 2024-04-24T23:49:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.493+0000 7f8a15ffb640 1 -- 172.21.15.16:0/1943349710 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8a1c018020 con 0x7f8a2006d360 2024-04-24T23:49:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.493+0000 7f8a15ffb640 1 --2- 172.21.15.16:0/1943349710 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a0406be30 0x7f8a0406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.493+0000 7f8a15ffb640 1 -- 172.21.15.16:0/1943349710 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8a1c08ac60 con 0x7f8a2006d360 2024-04-24T23:49:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.493+0000 7f8a24f3d640 1 --2- 172.21.15.16:0/1943349710 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a0406be30 0x7f8a0406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.494+0000 7f8a24f3d640 1 --2- 172.21.15.16:0/1943349710 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a0406be30 0x7f8a0406e2f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8a2015bf30 tx=0x7f8a10008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:51.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.498+0000 7f8a15ffb640 1 -- 172.21.15.16:0/1943349710 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8a1c01f030 con 0x7f8a2006d360 2024-04-24T23:49:51.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.684+0000 7f8a279a2640 1 -- 172.21.15.16:0/1943349710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8a20161730 con 0x7f8a2006d360 2024-04-24T23:49:51.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.684+0000 7f8a15ffb640 1 -- 172.21.15.16:0/1943349710 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f8a1c08a020 con 0x7f8a2006d360 2024-04-24T23:49:51.684 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.687+0000 7f8a279a2640 1 -- 172.21.15.16:0/1943349710 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a0406be30 msgr2=0x7f8a0406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.687+0000 7f8a279a2640 1 --2- 172.21.15.16:0/1943349710 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a0406be30 0x7f8a0406e2f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8a2015bf30 tx=0x7f8a10008040 comp rx=0 tx=0).stop 2024-04-24T23:49:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.687+0000 7f8a279a2640 1 -- 172.21.15.16:0/1943349710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a2006d360 msgr2=0x7f8a20158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.687+0000 7f8a279a2640 1 --2- 172.21.15.16:0/1943349710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a2006d360 0x7f8a20158350 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c00b3f0 tx=0x7f8a1c00b8b0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.687+0000 7f8a279a2640 1 -- 172.21.15.16:0/1943349710 shutdown_connections 2024-04-24T23:49:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.687+0000 7f8a279a2640 1 --2- 172.21.15.16:0/1943349710 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a0406be30 0x7f8a0406e2f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.687+0000 7f8a279a2640 1 --2- 172.21.15.16:0/1943349710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a2015dff0 0x7f8a20160a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.687+0000 7f8a279a2640 1 --2- 172.21.15.16:0/1943349710 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a200af980 0x7f8a20148b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.687+0000 7f8a279a2640 1 --2- 172.21.15.16:0/1943349710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a2006d360 0x7f8a20158350 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.687+0000 7f8a279a2640 1 -- 172.21.15.16:0/1943349710 >> 172.21.15.16:0/1943349710 conn(0x7f8a200affc0 msgr2=0x7f8a2015c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.687+0000 7f8a279a2640 1 -- 172.21.15.16:0/1943349710 shutdown_connections 2024-04-24T23:49:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.687+0000 7f8a279a2640 1 -- 172.21.15.16:0/1943349710 wait complete. 2024-04-24T23:49:51.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:49:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.799+0000 7efd00df9640 1 Processor -- start 2024-04-24T23:49:51.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.799+0000 7efd00df9640 1 -- start start 2024-04-24T23:49:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.800+0000 7efd00df9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7efcfc15ed10 0x7efcfc15e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.800+0000 7efd00df9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcfc06bcd0 con 0x7efcfc0cff10 2024-04-24T23:49:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.800+0000 7efd00df9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcfc06bfb0 con 0x7efcfc15e930 2024-04-24T23:49:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.800+0000 7efd00df9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7efcfc0d03a0 con 0x7efcfc15ed10 2024-04-24T23:49:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.800+0000 7efcfad9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efcfc0cff10 0x7efcfc0d1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42316/0 (socket says 172.21.15.16:42316) 2024-04-24T23:49:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcfad9d640 1 -- 172.21.15.16:0/3956863500 learned_addr learned my addr 172.21.15.16:0/3956863500 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf9d9b640 1 --2- 172.21.15.16:0/3956863500 >> v2:172.21.15.156:3300/0 conn(0x7efcfc15ed10 0x7efcfc15e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 -- 172.21.15.16:0/3956863500 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2443572549 0 0) 0x7efcfc06bcd0 con 0x7efcfc0cff10 2024-04-24T23:49:51.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 -- 172.21.15.16:0/3956863500 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efce4003520 con 0x7efcfc0cff10 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 -- 172.21.15.16:0/3956863500 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2708149021 0 0) 0x7efce4003520 con 0x7efcfc0cff10 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 -- 172.21.15.16:0/3956863500 >> v2:172.21.15.156:3300/0 conn(0x7efcfc15ed10 msgr2=0x7efcfc15e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 --2- 172.21.15.16:0/3956863500 >> v2:172.21.15.156:3300/0 conn(0x7efcfc15ed10 0x7efcfc15e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 -- 172.21.15.16:0/3956863500 >> v1:172.21.15.156:6789/0 conn(0x7efcfc15e930 legacy=0x7efcfc15dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 -- 172.21.15.16:0/3956863500 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcfc06c290 con 0x7efcfc0cff10 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 -- 172.21.15.16:0/3956863500 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7efcf4001670 con 0x7efcfc0cff10 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 -- 172.21.15.16:0/3956863500 >> v1:172.21.15.16:6789/0 conn(0x7efcfc0cff10 legacy=0x7efcfc0d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 --2- 172.21.15.16:0/3956863500 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efce40039b0 0x7efce4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 --2- 172.21.15.16:0/3956863500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efce40043b0 0x7efce4004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 --2- 172.21.15.16:0/3956863500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4005080 0x7efce400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 -- 172.21.15.16:0/3956863500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efcfc0d03a0 con 0x7efce4005080 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 -- 172.21.15.16:0/3956863500 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efce400bfe0 con 0x7efce40043b0 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf959a640 1 -- 172.21.15.16:0/3956863500 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7efce400c1e0 con 0x7efce40039b0 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcfad9d640 1 --2- 172.21.15.16:0/3956863500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4005080 0x7efce400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcf9d9b640 1 --2- 172.21.15.16:0/3956863500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efce40043b0 0x7efce4004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.801+0000 7efcfa59c640 1 --2- 172.21.15.16:0/3956863500 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efce40039b0 0x7efce4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.802+0000 7efcfad9d640 1 -- 172.21.15.16:0/3956863500 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efce40039b0 msgr2=0x7efce4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:51.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.802+0000 7efcfad9d640 1 --2- 172.21.15.16:0/3956863500 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efce40039b0 0x7efce4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.802+0000 7efcfad9d640 1 -- 172.21.15.16:0/3956863500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efce40043b0 msgr2=0x7efce4004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:51.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.802+0000 7efcfad9d640 1 --2- 172.21.15.16:0/3956863500 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efce40043b0 0x7efce4004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.802+0000 7efcfad9d640 1 -- 172.21.15.16:0/3956863500 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efce4003520 con 0x7efce4005080 2024-04-24T23:49:51.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.802+0000 7efcfad9d640 1 -- 172.21.15.16:0/3956863500 reap_dead start 2024-04-24T23:49:51.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.802+0000 7efcfad9d640 1 --2- 172.21.15.16:0/3956863500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4005080 0x7efce400b9c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efcf400c240 tx=0x7efce4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e96fea3dc27530 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:51.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.802+0000 7efcf959a640 1 -- 172.21.15.16:0/3956863500 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4007d10 con 0x7efce4005080 2024-04-24T23:49:51.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.802+0000 7efcf959a640 1 -- 172.21.15.16:0/3956863500 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf40058d0 con 0x7efce4005080 2024-04-24T23:49:51.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.802+0000 7efcf959a640 1 -- 172.21.15.16:0/3956863500 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf4005ca0 con 0x7efce4005080 2024-04-24T23:49:51.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.803+0000 7efd00df9640 1 -- 172.21.15.16:0/3956863500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4005080 msgr2=0x7efce400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.803+0000 7efd00df9640 1 --2- 172.21.15.16:0/3956863500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4005080 0x7efce400b9c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7efcf400c240 tx=0x7efce4005830 comp rx=0 tx=0).stop 2024-04-24T23:49:51.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.803+0000 7efd00df9640 1 -- 172.21.15.16:0/3956863500 shutdown_connections 2024-04-24T23:49:51.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.803+0000 7efd00df9640 1 --2- 172.21.15.16:0/3956863500 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4005080 0x7efce400b9c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.803+0000 7efd00df9640 1 -- 172.21.15.16:0/3956863500 >> 172.21.15.16:0/3956863500 conn(0x7efcfc0d4050 msgr2=0x7efcfc0d4460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:51.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.803+0000 7efd00df9640 1 -- 172.21.15.16:0/3956863500 shutdown_connections 2024-04-24T23:49:51.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.803+0000 7efd00df9640 1 -- 172.21.15.16:0/3956863500 wait complete. 2024-04-24T23:49:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.804+0000 7efd00df9640 1 Processor -- start 2024-04-24T23:49:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.804+0000 7efd00df9640 1 -- start start 2024-04-24T23:49:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.805+0000 7efd00df9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4005080 0x7efcfc159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.805+0000 7efd00df9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcfc0c8330 0x7efcfc160f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.805+0000 7efd00df9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcfc0c9e60 0x7efcfc0ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.805+0000 7efd00df9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7efcfc06b5a0 con 0x7efce4005080 2024-04-24T23:49:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.805+0000 7efd00df9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efcfc06c850 con 0x7efcfc0c8330 2024-04-24T23:49:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.805+0000 7efd00df9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7efcfc06b720 con 0x7efcfc0c9e60 2024-04-24T23:49:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.805+0000 7efcfa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4005080 0x7efcfc159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.805+0000 7efcfa59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4005080 0x7efcfc159780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40746/0 (socket says 172.21.15.16:40746) 2024-04-24T23:49:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.805+0000 7efcfa59c640 1 -- 172.21.15.16:0/998981542 learned_addr learned my addr 172.21.15.16:0/998981542 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efcf9d9b640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcfc0c8330 0x7efcfc160f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efcfad9d640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcfc0c9e60 0x7efcfc0ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efcfa59c640 1 -- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcfc0c9e60 msgr2=0x7efcfc0ca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efcfa59c640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcfc0c9e60 0x7efcfc0ca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efcfa59c640 1 -- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcfc0c8330 msgr2=0x7efcfc160f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efcfa59c640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcfc0c8330 0x7efcfc160f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efcfa59c640 1 -- 172.21.15.16:0/998981542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcfc1f3b90 con 0x7efce4005080 2024-04-24T23:49:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efcf9d9b640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcfc0c8330 0x7efcfc160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efcfad9d640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcfc0c9e60 0x7efcfc0ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efcfa59c640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4005080 0x7efcfc159780 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7efcec00fb60 tx=0x7efcec00ff60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efceaffd640 1 -- 172.21.15.16:0/998981542 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcec003b80 con 0x7efce4005080 2024-04-24T23:49:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efceaffd640 1 -- 172.21.15.16:0/998981542 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcec018070 con 0x7efce4005080 2024-04-24T23:49:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efd00df9640 1 -- 172.21.15.16:0/998981542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcfc0c8900 con 0x7efce4005080 2024-04-24T23:49:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efceaffd640 1 -- 172.21.15.16:0/998981542 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcec00d8e0 con 0x7efce4005080 2024-04-24T23:49:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.806+0000 7efd00df9640 1 -- 172.21.15.16:0/998981542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcfc0c8ec0 con 0x7efce4005080 2024-04-24T23:49:51.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.807+0000 7efd00df9640 1 -- 172.21.15.16:0/998981542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcfc1f3d60 con 0x7efce4005080 2024-04-24T23:49:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.807+0000 7efceaffd640 1 -- 172.21.15.16:0/998981542 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7efcec00da80 con 0x7efce4005080 2024-04-24T23:49:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.808+0000 7efceaffd640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efcdc06be30 0x7efcdc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.808+0000 7efceaffd640 1 -- 172.21.15.16:0/998981542 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7efcec08b540 con 0x7efce4005080 2024-04-24T23:49:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.808+0000 7efcf9d9b640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efcdc06be30 0x7efcdc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.809+0000 7efcf9d9b640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efcdc06be30 0x7efcdc06e2f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efcf00093d0 tx=0x7efcf0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:51.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:51.812+0000 7efceaffd640 1 -- 172.21.15.16:0/998981542 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7efcec059ae0 con 0x7efce4005080 2024-04-24T23:49:52.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.010+0000 7efd00df9640 1 -- 172.21.15.16:0/998981542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7efcfc06dfe0 con 0x7efce4005080 2024-04-24T23:49:52.010 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.010+0000 7efceaffd640 1 -- 172.21.15.16:0/998981542 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7efcec0593e0 con 0x7efce4005080 2024-04-24T23:49:52.010 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-04-24T23:49:52.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.012+0000 7efd00df9640 1 -- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efcdc06be30 msgr2=0x7efcdc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.012+0000 7efd00df9640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efcdc06be30 0x7efcdc06e2f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efcf00093d0 tx=0x7efcf0008040 comp rx=0 tx=0).stop 2024-04-24T23:49:52.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.012+0000 7efd00df9640 1 -- 172.21.15.16:0/998981542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4005080 msgr2=0x7efcfc159780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.012+0000 7efd00df9640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4005080 0x7efcfc159780 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7efcec00fb60 tx=0x7efcec00ff60 comp rx=0 tx=0).stop 2024-04-24T23:49:52.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.012+0000 7efd00df9640 1 -- 172.21.15.16:0/998981542 shutdown_connections 2024-04-24T23:49:52.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.012+0000 7efd00df9640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efcdc06be30 0x7efcdc06e2f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.012+0000 7efd00df9640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7efcfc0c9e60 0x7efcfc0ca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.012+0000 7efd00df9640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efcfc0c8330 0x7efcfc160f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.012+0000 7efd00df9640 1 --2- 172.21.15.16:0/998981542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efce4005080 0x7efcfc159780 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.012+0000 7efd00df9640 1 -- 172.21.15.16:0/998981542 >> 172.21.15.16:0/998981542 conn(0x7efcfc0d4050 msgr2=0x7efcfc0b7040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:52.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.012+0000 7efd00df9640 1 -- 172.21.15.16:0/998981542 shutdown_connections 2024-04-24T23:49:52.013 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.012+0000 7efd00df9640 1 -- 172.21.15.16:0/998981542 wait complete. 2024-04-24T23:49:52.023 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-04-24T23:49:52.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:52.023 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T23:49:52.048 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:49:52.048 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:49:52.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:52.048 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T23:49:52.104 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:49:52.104 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:49:52.123 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:49:52.123 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:49:52.182 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:52.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.284+0000 7f61d3163640 1 Processor -- start 2024-04-24T23:49:52.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.284+0000 7f61d3163640 1 -- start start 2024-04-24T23:49:52.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.285+0000 7f61d3163640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f61cc1586c0 0x7f61cc0b3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.285+0000 7f61d3163640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f61cc06c800 0x7f61cc161bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.285+0000 7f61d3163640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61cc15e810 con 0x7f61cc0af200 2024-04-24T23:49:52.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.285+0000 7f61d3163640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f61cc158aa0 con 0x7f61cc06c800 2024-04-24T23:49:52.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.285+0000 7f61d3163640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f61cc15eaf0 con 0x7f61cc1586c0 2024-04-24T23:49:52.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.285+0000 7f61d1700640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f61cc06c800 0x7f61cc161bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.285+0000 7f61d1700640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f61cc06c800 0x7f61cc161bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40754/0 (socket says 172.21.15.16:40754) 2024-04-24T23:49:52.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.285+0000 7f61d1700640 1 -- 172.21.15.16:0/819334122 learned_addr learned my addr 172.21.15.16:0/819334122 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:52.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.286+0000 7f61d0eff640 1 --2- 172.21.15.16:0/819334122 >> v2:172.21.15.156:3300/0 conn(0x7f61cc1586c0 0x7f61cc0b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.286+0000 7f61d1700640 1 -- 172.21.15.16:0/819334122 >> v2:172.21.15.156:3300/0 conn(0x7f61cc1586c0 msgr2=0x7f61cc0b3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:52.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.286+0000 7f61d1700640 1 --2- 172.21.15.16:0/819334122 >> v2:172.21.15.156:3300/0 conn(0x7f61cc1586c0 0x7f61cc0b3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.286+0000 7f61d1700640 1 -- 172.21.15.16:0/819334122 >> v1:172.21.15.156:6789/0 conn(0x7f61cc0af200 legacy=0x7f61cc06c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:52.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.286+0000 7f61d1700640 1 -- 172.21.15.16:0/819334122 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61cc15edd0 con 0x7f61cc06c800 2024-04-24T23:49:52.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.286+0000 7f61d0eff640 1 --2- 172.21.15.16:0/819334122 >> v2:172.21.15.156:3300/0 conn(0x7f61cc1586c0 0x7f61cc0b3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:52.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.286+0000 7f61d1700640 1 --2- 172.21.15.16:0/819334122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc06c800 0x7f61cc161bf0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f61cc06cc80 tx=0x7f61c402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6355a64d8add690a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:52.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.287+0000 7f61cb7fe640 1 -- 172.21.15.16:0/819334122 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c4036070 con 0x7f61cc06c800 2024-04-24T23:49:52.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.287+0000 7f61cb7fe640 1 -- 172.21.15.16:0/819334122 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61c4031600 con 0x7f61cc06c800 2024-04-24T23:49:52.287 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.287+0000 7f61cb7fe640 1 -- 172.21.15.16:0/819334122 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61c40319d0 con 0x7f61cc06c800 2024-04-24T23:49:52.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.287+0000 7f61d3163640 1 -- 172.21.15.16:0/819334122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc06c800 msgr2=0x7f61cc161bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.287+0000 7f61d3163640 1 --2- 172.21.15.16:0/819334122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc06c800 0x7f61cc161bf0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f61cc06cc80 tx=0x7f61c402a6f0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.287+0000 7f61d3163640 1 -- 172.21.15.16:0/819334122 shutdown_connections 2024-04-24T23:49:52.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.287+0000 7f61d3163640 1 --2- 172.21.15.16:0/819334122 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc06c800 0x7f61cc161bf0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.287+0000 7f61d3163640 1 --2- 172.21.15.16:0/819334122 >> v2:172.21.15.156:3300/0 conn(0x7f61cc1586c0 0x7f61cc0b3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.287+0000 7f61d3163640 1 -- 172.21.15.16:0/819334122 >> 172.21.15.16:0/819334122 conn(0x7f61cc0cc8d0 msgr2=0x7f61cc0ccce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:52.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.287+0000 7f61d3163640 1 -- 172.21.15.16:0/819334122 shutdown_connections 2024-04-24T23:49:52.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.287+0000 7f61d3163640 1 -- 172.21.15.16:0/819334122 wait complete. 2024-04-24T23:49:52.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.288+0000 7f61d3163640 1 Processor -- start 2024-04-24T23:49:52.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.288+0000 7f61d3163640 1 -- start start 2024-04-24T23:49:52.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.289+0000 7f61d3163640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61cc06c800 0x7f61cc071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.291 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.289+0000 7f61d3163640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc0af200 0x7f61cc158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.289+0000 7f61d3163640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61cc1586c0 0x7f61cc072960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.289+0000 7f61d3163640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc0ad8e0 con 0x7f61cc0af200 2024-04-24T23:49:52.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.289+0000 7f61d3163640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f61cc0af640 con 0x7f61cc1586c0 2024-04-24T23:49:52.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.289+0000 7f61d3163640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f61cc0ada60 con 0x7f61cc06c800 2024-04-24T23:49:52.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.289+0000 7f61cbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc0af200 0x7f61cc158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.289+0000 7f61cbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc0af200 0x7f61cc158c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40762/0 (socket says 172.21.15.16:40762) 2024-04-24T23:49:52.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.289+0000 7f61cbfff640 1 -- 172.21.15.16:0/3710817432 learned_addr learned my addr 172.21.15.16:0/3710817432 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:52.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.290+0000 7f61d1700640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61cc1586c0 0x7f61cc072960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.290+0000 7f61d0eff640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61cc06c800 0x7f61cc071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.290+0000 7f61cbfff640 1 -- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61cc06c800 msgr2=0x7f61cc071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.290+0000 7f61cbfff640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61cc06c800 0x7f61cc071030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.290+0000 7f61cbfff640 1 -- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61cc1586c0 msgr2=0x7f61cc072960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.292 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.290+0000 7f61cbfff640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61cc1586c0 0x7f61cc072960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.290+0000 7f61cbfff640 1 -- 172.21.15.16:0/3710817432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61cc0731d0 con 0x7f61cc0af200 2024-04-24T23:49:52.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.290+0000 7f61d0eff640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61cc06c800 0x7f61cc071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:52.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.290+0000 7f61cbfff640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc0af200 0x7f61cc158c30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f61b800c430 tx=0x7f61b800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:52.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.290+0000 7f61c97fa640 1 -- 172.21.15.16:0/3710817432 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b8011790 con 0x7f61cc0af200 2024-04-24T23:49:52.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.290+0000 7f61c97fa640 1 -- 172.21.15.16:0/3710817432 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61b8017070 con 0x7f61cc0af200 2024-04-24T23:49:52.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.290+0000 7f61c97fa640 1 -- 172.21.15.16:0/3710817432 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61b8011bb0 con 0x7f61cc0af200 2024-04-24T23:49:52.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.290+0000 7f61d3163640 1 -- 172.21.15.16:0/3710817432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61cc1543a0 con 0x7f61cc0af200 2024-04-24T23:49:52.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.290+0000 7f61d3163640 1 -- 172.21.15.16:0/3710817432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61cc154960 con 0x7f61cc0af200 2024-04-24T23:49:52.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.291+0000 7f61d3163640 1 -- 172.21.15.16:0/3710817432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61cc154b60 con 0x7f61cc0af200 2024-04-24T23:49:52.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.292+0000 7f61c97fa640 1 -- 172.21.15.16:0/3710817432 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f61b801c460 con 0x7f61cc0af200 2024-04-24T23:49:52.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.292+0000 7f61c97fa640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f61b006be30 0x7f61b006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.292+0000 7f61c97fa640 1 -- 172.21.15.16:0/3710817432 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f61b808b160 con 0x7f61cc0af200 2024-04-24T23:49:52.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.292+0000 7f61d0eff640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f61b006be30 0x7f61b006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.294 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.293+0000 7f61d0eff640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f61b006be30 0x7f61b006e2f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f61c0006fd0 tx=0x7f61c0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:52.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.296+0000 7f61c97fa640 1 -- 172.21.15.16:0/3710817432 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f61cc154b60 con 0x7f61cc0af200 2024-04-24T23:49:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.482+0000 7f61d3163640 1 -- 172.21.15.16:0/3710817432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f61cc0735e0 con 0x7f61cc0af200 2024-04-24T23:49:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.484+0000 7f61c97fa640 1 -- 172.21.15.16:0/3710817432 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7f61b8059710 con 0x7f61cc0af200 2024-04-24T23:49:52.483 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:52.484 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.485+0000 7f61d3163640 1 -- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f61b006be30 msgr2=0x7f61b006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.485+0000 7f61d3163640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f61b006be30 0x7f61b006e2f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f61c0006fd0 tx=0x7f61c0008040 comp rx=0 tx=0).stop 2024-04-24T23:49:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.485+0000 7f61d3163640 1 -- 172.21.15.16:0/3710817432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc0af200 msgr2=0x7f61cc158c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.485+0000 7f61d3163640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc0af200 0x7f61cc158c30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f61b800c430 tx=0x7f61b800c8f0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.486+0000 7f61d3163640 1 -- 172.21.15.16:0/3710817432 shutdown_connections 2024-04-24T23:49:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.486+0000 7f61d3163640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f61b006be30 0x7f61b006e2f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.486+0000 7f61d3163640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61cc1586c0 0x7f61cc072960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.486+0000 7f61d3163640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61cc0af200 0x7f61cc158c30 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.486+0000 7f61d3163640 1 --2- 172.21.15.16:0/3710817432 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f61cc06c800 0x7f61cc071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.486+0000 7f61d3163640 1 -- 172.21.15.16:0/3710817432 >> 172.21.15.16:0/3710817432 conn(0x7f61cc0cc8d0 msgr2=0x7f61cc162140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.486+0000 7f61d3163640 1 -- 172.21.15.16:0/3710817432 shutdown_connections 2024-04-24T23:49:52.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.486+0000 7f61d3163640 1 -- 172.21.15.16:0/3710817432 wait complete. 2024-04-24T23:49:52.497 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:52.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.601+0000 7f7ccdde9640 1 Processor -- start 2024-04-24T23:49:52.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.601+0000 7f7ccdde9640 1 -- start start 2024-04-24T23:49:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.602+0000 7f7ccdde9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cc80d12d0 0x7f7cc80d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.602+0000 7f7ccdde9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7cc815e5c0 0x7f7cc80d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.602+0000 7f7ccdde9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cc815ec30 con 0x7f7cc80cd790 2024-04-24T23:49:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.602+0000 7f7ccdde9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f7cc815f4d0 con 0x7f7cc815e5c0 2024-04-24T23:49:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.602+0000 7f7ccdde9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc815ef10 con 0x7f7cc80d12d0 2024-04-24T23:49:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.602+0000 7f7cc6ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7cc80cd790 0x7f7cc80c8580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42318/0 (socket says 172.21.15.16:42318) 2024-04-24T23:49:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.602+0000 7f7cc6ffd640 1 -- 172.21.15.16:0/4058745938 learned_addr learned my addr 172.21.15.16:0/4058745938 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc77fe640 1 --2- 172.21.15.16:0/4058745938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cc80d12d0 0x7f7cc80d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc7fff640 1 --2- 172.21.15.16:0/4058745938 >> v2:172.21.15.156:3300/0 conn(0x7f7cc815e5c0 0x7f7cc80d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 -- 172.21.15.16:0/4058745938 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3599988570 0 0) 0x7f7cc815ec30 con 0x7f7cc80cd790 2024-04-24T23:49:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 -- 172.21.15.16:0/4058745938 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cb0003520 con 0x7f7cc80cd790 2024-04-24T23:49:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 -- 172.21.15.16:0/4058745938 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1259776089 0 0) 0x7f7cb0003520 con 0x7f7cc80cd790 2024-04-24T23:49:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 -- 172.21.15.16:0/4058745938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cc80d12d0 msgr2=0x7f7cc80d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 --2- 172.21.15.16:0/4058745938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cc80d12d0 0x7f7cc80d16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 -- 172.21.15.16:0/4058745938 >> v2:172.21.15.156:3300/0 conn(0x7f7cc815e5c0 msgr2=0x7f7cc80d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 --2- 172.21.15.16:0/4058745938 >> v2:172.21.15.156:3300/0 conn(0x7f7cc815e5c0 0x7f7cc80d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 -- 172.21.15.16:0/4058745938 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc815f1f0 con 0x7f7cc80cd790 2024-04-24T23:49:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 -- 172.21.15.16:0/4058745938 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f7cbc001670 con 0x7f7cc80cd790 2024-04-24T23:49:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 -- 172.21.15.16:0/4058745938 >> v1:172.21.15.16:6789/0 conn(0x7f7cc80cd790 legacy=0x7f7cc80c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 --2- 172.21.15.16:0/4058745938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cc80d12d0 0x7f7cb00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 --2- 172.21.15.16:0/4058745938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0003f30 0x7f7cb0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.613 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 --2- 172.21.15.16:0/4058745938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cb0004d40 0x7f7cb00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 -- 172.21.15.16:0/4058745938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc815f4d0 con 0x7f7cb0003f30 2024-04-24T23:49:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 -- 172.21.15.16:0/4058745938 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc815ef10 con 0x7f7cb0004d40 2024-04-24T23:49:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc67fc640 1 -- 172.21.15.16:0/4058745938 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7cb000baa0 con 0x7f7cc80d12d0 2024-04-24T23:49:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc6ffd640 1 --2- 172.21.15.16:0/4058745938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0003f30 0x7f7cb0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc77fe640 1 --2- 172.21.15.16:0/4058745938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cc80d12d0 0x7f7cb00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc7fff640 1 --2- 172.21.15.16:0/4058745938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cb0004d40 0x7f7cb00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc6ffd640 1 -- 172.21.15.16:0/4058745938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cc80d12d0 msgr2=0x7f7cb00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc6ffd640 1 --2- 172.21.15.16:0/4058745938 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cc80d12d0 0x7f7cb00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc6ffd640 1 -- 172.21.15.16:0/4058745938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cb0004d40 msgr2=0x7f7cb00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc6ffd640 1 --2- 172.21.15.16:0/4058745938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cb0004d40 0x7f7cb00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc6ffd640 1 -- 172.21.15.16:0/4058745938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cb0003520 con 0x7f7cb0003f30 2024-04-24T23:49:52.614 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.603+0000 7f7cc7fff640 1 --2- 172.21.15.16:0/4058745938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cb0004d40 0x7f7cb00051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.604+0000 7f7cc6ffd640 1 --2- 172.21.15.16:0/4058745938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0003f30 0x7f7cb0004800 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc007fd0 tx=0x7f7cb0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cdd6bd7a63320e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.604+0000 7f7cc67fc640 1 -- 172.21.15.16:0/4058745938 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc007d50 con 0x7f7cb0003f30 2024-04-24T23:49:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.604+0000 7f7cc67fc640 1 -- 172.21.15.16:0/4058745938 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cbc005ad0 con 0x7f7cb0003f30 2024-04-24T23:49:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.604+0000 7f7cc67fc640 1 -- 172.21.15.16:0/4058745938 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc007070 con 0x7f7cb0003f30 2024-04-24T23:49:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.605+0000 7f7ccdde9640 1 -- 172.21.15.16:0/4058745938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0003f30 msgr2=0x7f7cb0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.605+0000 7f7ccdde9640 1 --2- 172.21.15.16:0/4058745938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0003f30 0x7f7cb0004800 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc007fd0 tx=0x7f7cb0005830 comp rx=0 tx=0).stop 2024-04-24T23:49:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.605+0000 7f7cc7fff640 1 -- 172.21.15.16:0/4058745938 reap_dead start 2024-04-24T23:49:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.605+0000 7f7ccdde9640 1 -- 172.21.15.16:0/4058745938 shutdown_connections 2024-04-24T23:49:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.605+0000 7f7ccdde9640 1 -- 172.21.15.16:0/4058745938 >> 172.21.15.16:0/4058745938 conn(0x7f7cc8072e70 msgr2=0x7f7cc8073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.605+0000 7f7ccdde9640 1 -- 172.21.15.16:0/4058745938 shutdown_connections 2024-04-24T23:49:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.605+0000 7f7ccdde9640 1 -- 172.21.15.16:0/4058745938 wait complete. 2024-04-24T23:49:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.606+0000 7f7ccdde9640 1 Processor -- start 2024-04-24T23:49:52.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.606+0000 7f7ccdde9640 1 -- start start 2024-04-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7ccdde9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0003f30 0x7f7cc8148270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7ccdde9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cc80b04e0 0x7f7cc80c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7ccdde9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cc8144be0 0x7f7cc80ce790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7ccdde9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc80d1830 con 0x7f7cb0003f30 2024-04-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7ccdde9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc80d16b0 con 0x7f7cc80b04e0 2024-04-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7ccdde9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc80d19b0 con 0x7f7cc8144be0 2024-04-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7cc77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0003f30 0x7f7cc8148270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7cc77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0003f30 0x7f7cc8148270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40770/0 (socket says 172.21.15.16:40770) 2024-04-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7cc77fe640 1 -- 172.21.15.16:0/2411642370 learned_addr learned my addr 172.21.15.16:0/2411642370 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7cc7fff640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cc8144be0 0x7f7cc80ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7cc6ffd640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cc80b04e0 0x7f7cc80c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7cc77fe640 1 -- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cc8144be0 msgr2=0x7f7cc80ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7cc77fe640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cc8144be0 0x7f7cc80ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7cc77fe640 1 -- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cc80b04e0 msgr2=0x7f7cc80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7cc77fe640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cc80b04e0 0x7f7cc80c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7cc77fe640 1 -- 172.21.15.16:0/2411642370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc81f7b10 con 0x7f7cb0003f30 2024-04-24T23:49:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7cc7fff640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cc8144be0 0x7f7cc80ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.607+0000 7f7cc6ffd640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cc80b04e0 0x7f7cc80c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.608+0000 7f7cc77fe640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0003f30 0x7f7cc8148270 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7cb80119d0 tx=0x7f7cb8011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.608+0000 7f7caffff640 1 -- 172.21.15.16:0/2411642370 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb800faf0 con 0x7f7cb0003f30 2024-04-24T23:49:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.608+0000 7f7caffff640 1 -- 172.21.15.16:0/2411642370 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cb800fc90 con 0x7f7cb0003f30 2024-04-24T23:49:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.608+0000 7f7caffff640 1 -- 172.21.15.16:0/2411642370 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb8014440 con 0x7f7cb0003f30 2024-04-24T23:49:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.608+0000 7f7ccdde9640 1 -- 172.21.15.16:0/2411642370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc81f7df0 con 0x7f7cb0003f30 2024-04-24T23:49:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.608+0000 7f7ccdde9640 1 -- 172.21.15.16:0/2411642370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc81446c0 con 0x7f7cb0003f30 2024-04-24T23:49:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.609+0000 7f7ccdde9640 1 -- 172.21.15.16:0/2411642370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cc8068f90 con 0x7f7cb0003f30 2024-04-24T23:49:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.609+0000 7f7caffff640 1 -- 172.21.15.16:0/2411642370 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7cb8012040 con 0x7f7cb0003f30 2024-04-24T23:49:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.610+0000 7f7caffff640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7cb406be60 0x7f7cb406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.610+0000 7f7caffff640 1 -- 172.21.15.16:0/2411642370 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7cb808b800 con 0x7f7cb0003f30 2024-04-24T23:49:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.610+0000 7f7cc6ffd640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7cb406be60 0x7f7cb406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.610+0000 7f7cc6ffd640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7cb406be60 0x7f7cb406e320 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc033000 tx=0x7f7cbc009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:52.618 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.614+0000 7f7caffff640 1 -- 172.21.15.16:0/2411642370 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7cb8019030 con 0x7f7cb0003f30 2024-04-24T23:49:52.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.795+0000 7f7ccdde9640 1 -- 172.21.15.16:0/2411642370 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7cc815f680 con 0x7f7cb0003f30 2024-04-24T23:49:52.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.796+0000 7f7caffff640 1 -- 172.21.15.16:0/2411642370 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f7cb8059db0 con 0x7f7cb0003f30 2024-04-24T23:49:52.796 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:52.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.798+0000 7f7ccdde9640 1 -- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7cb406be60 msgr2=0x7f7cb406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.798+0000 7f7ccdde9640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7cb406be60 0x7f7cb406e320 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc033000 tx=0x7f7cbc009030 comp rx=0 tx=0).stop 2024-04-24T23:49:52.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.798+0000 7f7ccdde9640 1 -- 172.21.15.16:0/2411642370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0003f30 msgr2=0x7f7cc8148270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.798+0000 7f7ccdde9640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0003f30 0x7f7cc8148270 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f7cb80119d0 tx=0x7f7cb8011e90 comp rx=0 tx=0).stop 2024-04-24T23:49:52.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.798+0000 7f7ccdde9640 1 -- 172.21.15.16:0/2411642370 shutdown_connections 2024-04-24T23:49:52.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.798+0000 7f7ccdde9640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7cb406be60 0x7f7cb406e320 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.798+0000 7f7ccdde9640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7cc8144be0 0x7f7cc80ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.798+0000 7f7ccdde9640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7cc80b04e0 0x7f7cc80c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.798+0000 7f7ccdde9640 1 --2- 172.21.15.16:0/2411642370 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cb0003f30 0x7f7cc8148270 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.798+0000 7f7ccdde9640 1 -- 172.21.15.16:0/2411642370 >> 172.21.15.16:0/2411642370 conn(0x7f7cc8072e70 msgr2=0x7f7cc8154790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.798+0000 7f7ccdde9640 1 -- 172.21.15.16:0/2411642370 shutdown_connections 2024-04-24T23:49:52.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.798+0000 7f7ccdde9640 1 -- 172.21.15.16:0/2411642370 wait complete. 2024-04-24T23:49:52.809 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:52.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.912+0000 7f376f438640 1 Processor -- start 2024-04-24T23:49:52.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.912+0000 7f376f438640 1 -- start start 2024-04-24T23:49:52.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.913+0000 7f376f438640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f376806c080 0x7f37680ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.913+0000 7f376f438640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37680aebb0 con 0x7f3768161510 2024-04-24T23:49:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.913+0000 7f376f438640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37680adfa0 con 0x7f37680b6230 2024-04-24T23:49:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.913+0000 7f376f438640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f37680ae280 con 0x7f376806c080 2024-04-24T23:49:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.913+0000 7f376d1d4640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3768161510 0x7f376806b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42334/0 (socket says 172.21.15.16:42334) 2024-04-24T23:49:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.913+0000 7f376d1d4640 1 -- 172.21.15.16:0/1523861063 learned_addr learned my addr 172.21.15.16:0/1523861063 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.913+0000 7f376c9d3640 1 --2- 172.21.15.16:0/1523861063 >> v2:172.21.15.16:3300/0 conn(0x7f376806c080 0x7f37680ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.914+0000 7f3767fff640 1 -- 172.21.15.16:0/1523861063 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1158884508 0 0) 0x7f37680aebb0 con 0x7f3768161510 2024-04-24T23:49:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.914+0000 7f3767fff640 1 -- 172.21.15.16:0/1523861063 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3750003520 con 0x7f3768161510 2024-04-24T23:49:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.914+0000 7f376c9d3640 1 -- 172.21.15.16:0/1523861063 >> v1:172.21.15.156:6789/0 conn(0x7f37680b6230 legacy=0x7f37680b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.914+0000 7f376c9d3640 1 -- 172.21.15.16:0/1523861063 >> v1:172.21.15.16:6789/0 conn(0x7f3768161510 legacy=0x7f376806b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.914+0000 7f376c9d3640 1 -- 172.21.15.16:0/1523861063 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37680ae560 con 0x7f376806c080 2024-04-24T23:49:52.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.915+0000 7f376c9d3640 1 --2- 172.21.15.16:0/1523861063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f376806c080 0x7f37680ad8e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f375c0097a0 tx=0x7f375c02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7d769b03f90e83 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.915+0000 7f3767fff640 1 -- 172.21.15.16:0/1523861063 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c037070 con 0x7f376806c080 2024-04-24T23:49:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.915+0000 7f3767fff640 1 -- 172.21.15.16:0/1523861063 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f375c032440 con 0x7f376806c080 2024-04-24T23:49:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.915+0000 7f3767fff640 1 -- 172.21.15.16:0/1523861063 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375c032810 con 0x7f376806c080 2024-04-24T23:49:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.915+0000 7f376f438640 1 -- 172.21.15.16:0/1523861063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f376806c080 msgr2=0x7f37680ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.915+0000 7f376f438640 1 --2- 172.21.15.16:0/1523861063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f376806c080 0x7f37680ad8e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f375c0097a0 tx=0x7f375c02a230 comp rx=0 tx=0).stop 2024-04-24T23:49:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.915+0000 7f376f438640 1 -- 172.21.15.16:0/1523861063 shutdown_connections 2024-04-24T23:49:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.915+0000 7f376f438640 1 --2- 172.21.15.16:0/1523861063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f376806c080 0x7f37680ad8e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.915+0000 7f376f438640 1 -- 172.21.15.16:0/1523861063 >> 172.21.15.16:0/1523861063 conn(0x7f3768072aa0 msgr2=0x7f3768072eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.915+0000 7f376f438640 1 -- 172.21.15.16:0/1523861063 shutdown_connections 2024-04-24T23:49:52.916 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.915+0000 7f376f438640 1 -- 172.21.15.16:0/1523861063 wait complete. 2024-04-24T23:49:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.916+0000 7f376f438640 1 Processor -- start 2024-04-24T23:49:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.917+0000 7f376f438640 1 -- start start 2024-04-24T23:49:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.917+0000 7f376f438640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f376806c080 0x7f37681f7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.918 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.917+0000 7f376f438640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f37680b6230 0x7f37680b5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.917+0000 7f376f438640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3768161510 0x7f37681415a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.917+0000 7f376f438640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f376806da90 con 0x7f376806c080 2024-04-24T23:49:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.917+0000 7f376f438640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f37680b6660 con 0x7f3768161510 2024-04-24T23:49:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.917+0000 7f376f438640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f376806dc10 con 0x7f37680b6230 2024-04-24T23:49:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.917+0000 7f376d1d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f376806c080 0x7f37681f7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.917+0000 7f376d1d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f376806c080 0x7f37681f7ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40784/0 (socket says 172.21.15.16:40784) 2024-04-24T23:49:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.917+0000 7f376d1d4640 1 -- 172.21.15.16:0/2623720061 learned_addr learned my addr 172.21.15.16:0/2623720061 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.918+0000 7f376d9d5640 1 --2- 172.21.15.16:0/2623720061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3768161510 0x7f37681415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.918+0000 7f376c9d3640 1 --2- 172.21.15.16:0/2623720061 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f37680b6230 0x7f37680b5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.918+0000 7f376d1d4640 1 -- 172.21.15.16:0/2623720061 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f37680b6230 msgr2=0x7f37680b5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.918+0000 7f376d1d4640 1 --2- 172.21.15.16:0/2623720061 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f37680b6230 0x7f37680b5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.918+0000 7f376d1d4640 1 -- 172.21.15.16:0/2623720061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3768161510 msgr2=0x7f37681415a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.918+0000 7f376d1d4640 1 --2- 172.21.15.16:0/2623720061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3768161510 0x7f37681415a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.918+0000 7f376d1d4640 1 -- 172.21.15.16:0/2623720061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3768152630 con 0x7f376806c080 2024-04-24T23:49:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.918+0000 7f376d1d4640 1 --2- 172.21.15.16:0/2623720061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f376806c080 0x7f37681f7ca0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f376000c400 tx=0x7f376000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.918+0000 7f3765ffb640 1 -- 172.21.15.16:0/2623720061 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760017070 con 0x7f376806c080 2024-04-24T23:49:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.918+0000 7f3765ffb640 1 -- 172.21.15.16:0/2623720061 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3760005ce0 con 0x7f376806c080 2024-04-24T23:49:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.918+0000 7f376f438640 1 -- 172.21.15.16:0/2623720061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37681447f0 con 0x7f376806c080 2024-04-24T23:49:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.918+0000 7f3765ffb640 1 -- 172.21.15.16:0/2623720061 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37600072d0 con 0x7f376806c080 2024-04-24T23:49:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.918+0000 7f376f438640 1 -- 172.21.15.16:0/2623720061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3768144db0 con 0x7f376806c080 2024-04-24T23:49:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.919+0000 7f376f438640 1 -- 172.21.15.16:0/2623720061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3768068f90 con 0x7f376806c080 2024-04-24T23:49:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.919+0000 7f3765ffb640 1 -- 172.21.15.16:0/2623720061 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f376001a070 con 0x7f376806c080 2024-04-24T23:49:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.920+0000 7f3765ffb640 1 --2- 172.21.15.16:0/2623720061 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f375406be30 0x7f375406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.920+0000 7f3765ffb640 1 -- 172.21.15.16:0/2623720061 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f376008a550 con 0x7f376806c080 2024-04-24T23:49:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.920+0000 7f376c9d3640 1 --2- 172.21.15.16:0/2623720061 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f375406be30 0x7f375406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.921+0000 7f376c9d3640 1 --2- 172.21.15.16:0/2623720061 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f375406be30 0x7f375406e2f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f375c034000 tx=0x7f375c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:52.924+0000 7f3765ffb640 1 -- 172.21.15.16:0/2623720061 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f37600584c0 con 0x7f376806c080 2024-04-24T23:49:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.107+0000 7f376f438640 1 -- 172.21.15.16:0/2623720061 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f376806c860 con 0x7f376806c080 2024-04-24T23:49:53.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.108+0000 7f3765ffb640 1 -- 172.21.15.16:0/2623720061 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f376005c7c0 con 0x7f376806c080 2024-04-24T23:49:53.108 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.110+0000 7f376f438640 1 -- 172.21.15.16:0/2623720061 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f375406be30 msgr2=0x7f375406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.110+0000 7f376f438640 1 --2- 172.21.15.16:0/2623720061 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f375406be30 0x7f375406e2f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f375c034000 tx=0x7f375c02e000 comp rx=0 tx=0).stop 2024-04-24T23:49:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.110+0000 7f376f438640 1 -- 172.21.15.16:0/2623720061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f376806c080 msgr2=0x7f37681f7ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.110+0000 7f376f438640 1 --2- 172.21.15.16:0/2623720061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f376806c080 0x7f37681f7ca0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f376000c400 tx=0x7f376000c8c0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.110+0000 7f376f438640 1 -- 172.21.15.16:0/2623720061 shutdown_connections 2024-04-24T23:49:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.110+0000 7f376f438640 1 --2- 172.21.15.16:0/2623720061 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f375406be30 0x7f375406e2f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.110 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.110+0000 7f376f438640 1 --2- 172.21.15.16:0/2623720061 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3768161510 0x7f37681415a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.110+0000 7f376f438640 1 --2- 172.21.15.16:0/2623720061 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f37680b6230 0x7f37680b5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.110+0000 7f376f438640 1 --2- 172.21.15.16:0/2623720061 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f376806c080 0x7f37681f7ca0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.110+0000 7f376f438640 1 -- 172.21.15.16:0/2623720061 >> 172.21.15.16:0/2623720061 conn(0x7f3768072aa0 msgr2=0x7f37680aedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.110+0000 7f376f438640 1 -- 172.21.15.16:0/2623720061 shutdown_connections 2024-04-24T23:49:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.110+0000 7f376f438640 1 -- 172.21.15.16:0/2623720061 wait complete. 2024-04-24T23:49:53.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:53.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.225+0000 7f8349528640 1 Processor -- start 2024-04-24T23:49:53.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.225+0000 7f8349528640 1 -- start start 2024-04-24T23:49:53.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.226+0000 7f8349528640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f834415e2f0 0x7f834406b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.226+0000 7f8349528640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f83440b75c0 0x7f83440b7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.226+0000 7f8349528640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f834415e6d0 con 0x7f83440d26e0 2024-04-24T23:49:53.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.226+0000 7f8349528640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f83440d2ca0 con 0x7f83440b75c0 2024-04-24T23:49:53.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.226+0000 7f8349528640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f83440d2f80 con 0x7f834415e2f0 2024-04-24T23:49:53.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.226+0000 7f83437fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f83440b75c0 0x7f83440b7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.226+0000 7f8342ffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f834415e2f0 0x7f834406b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.226+0000 7f8342ffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f834415e2f0 0x7f834406b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:57872/0 (socket says 172.21.15.16:57872) 2024-04-24T23:49:53.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.226+0000 7f83437fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f83440b75c0 0x7f83440b7a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:54698/0 (socket says 172.21.15.16:54698) 2024-04-24T23:49:53.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.226+0000 7f8342ffd640 1 -- 172.21.15.16:0/1865054788 learned_addr learned my addr 172.21.15.16:0/1865054788 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:53.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.226+0000 7f833bfff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f83440d26e0 0x7f834406b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:32794/0 (socket says 172.21.15.16:32794) 2024-04-24T23:49:53.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.227+0000 7f8342ffd640 1 -- 172.21.15.16:0/1865054788 >> v2:172.21.15.156:3300/0 conn(0x7f83440b75c0 msgr2=0x7f83440b7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:53.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.227+0000 7f8342ffd640 1 --2- 172.21.15.16:0/1865054788 >> v2:172.21.15.156:3300/0 conn(0x7f83440b75c0 0x7f83440b7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.227+0000 7f83427fc640 1 -- 172.21.15.16:0/1865054788 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2529806135 0 0) 0x7f834415e6d0 con 0x7f83440d26e0 2024-04-24T23:49:53.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.227+0000 7f8342ffd640 1 -- 172.21.15.16:0/1865054788 >> v1:172.21.15.156:6789/0 conn(0x7f83440d26e0 legacy=0x7f834406b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.227 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.227+0000 7f8342ffd640 1 -- 172.21.15.16:0/1865054788 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83440d3260 con 0x7f834415e2f0 2024-04-24T23:49:53.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.227+0000 7f8342ffd640 1 --2- 172.21.15.16:0/1865054788 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f834415e2f0 0x7f834406b410 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8334009870 tx=0x7f833402a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c67fd7facc88843 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:53.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.228+0000 7f83427fc640 1 -- 172.21.15.16:0/1865054788 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334037070 con 0x7f834415e2f0 2024-04-24T23:49:53.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.228+0000 7f83427fc640 1 -- 172.21.15.16:0/1865054788 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f833402ae50 con 0x7f834415e2f0 2024-04-24T23:49:53.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.228+0000 7f83427fc640 1 -- 172.21.15.16:0/1865054788 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334032860 con 0x7f834415e2f0 2024-04-24T23:49:53.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.228+0000 7f8349528640 1 -- 172.21.15.16:0/1865054788 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f834415e2f0 msgr2=0x7f834406b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.228+0000 7f8349528640 1 --2- 172.21.15.16:0/1865054788 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f834415e2f0 0x7f834406b410 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8334009870 tx=0x7f833402a300 comp rx=0 tx=0).stop 2024-04-24T23:49:53.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.228+0000 7f8349528640 1 -- 172.21.15.16:0/1865054788 shutdown_connections 2024-04-24T23:49:53.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.228+0000 7f8349528640 1 --2- 172.21.15.16:0/1865054788 >> v2:172.21.15.156:3300/0 conn(0x7f83440b75c0 0x7f83440b7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.228+0000 7f8349528640 1 --2- 172.21.15.16:0/1865054788 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f834415e2f0 0x7f834406b410 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.228+0000 7f8349528640 1 -- 172.21.15.16:0/1865054788 >> 172.21.15.16:0/1865054788 conn(0x7f83440d4850 msgr2=0x7f83440d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:53.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.228+0000 7f8349528640 1 -- 172.21.15.16:0/1865054788 shutdown_connections 2024-04-24T23:49:53.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.228+0000 7f8349528640 1 -- 172.21.15.16:0/1865054788 wait complete. 2024-04-24T23:49:53.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.229+0000 7f8349528640 1 Processor -- start 2024-04-24T23:49:53.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.229+0000 7f8349528640 1 -- start start 2024-04-24T23:49:53.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f8349528640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83440b75c0 0x7f83440d0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f8349528640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83440d26e0 0x7f83440cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f8349528640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f834415e2f0 0x7f83440cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f8349528640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f83440d0080 con 0x7f83440b75c0 2024-04-24T23:49:53.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f8349528640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f83440b70a0 con 0x7f834415e2f0 2024-04-24T23:49:53.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f8349528640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f834406c060 con 0x7f83440d26e0 2024-04-24T23:49:53.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f8342ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83440b75c0 0x7f83440d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f8342ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83440b75c0 0x7f83440d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40788/0 (socket says 172.21.15.16:40788) 2024-04-24T23:49:53.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f8342ffd640 1 -- 172.21.15.16:0/3144984347 learned_addr learned my addr 172.21.15.16:0/3144984347 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:53.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f83437fe640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f834415e2f0 0x7f83440cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f833bfff640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83440d26e0 0x7f83440cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f8342ffd640 1 -- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83440d26e0 msgr2=0x7f83440cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f8342ffd640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83440d26e0 0x7f83440cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f8342ffd640 1 -- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f834415e2f0 msgr2=0x7f83440cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f8342ffd640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f834415e2f0 0x7f83440cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.230+0000 7f8342ffd640 1 -- 172.21.15.16:0/3144984347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83440cb9d0 con 0x7f83440b75c0 2024-04-24T23:49:53.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.231+0000 7f83437fe640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f834415e2f0 0x7f83440cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:53.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.231+0000 7f833bfff640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83440d26e0 0x7f83440cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:53.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.231+0000 7f8342ffd640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83440b75c0 0x7f83440d0470 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f83340090b0 tx=0x7f83340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:53.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.231+0000 7f833b7fe640 1 -- 172.21.15.16:0/3144984347 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334041020 con 0x7f83440b75c0 2024-04-24T23:49:53.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.231+0000 7f833b7fe640 1 -- 172.21.15.16:0/3144984347 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f833403c890 con 0x7f83440b75c0 2024-04-24T23:49:53.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.231+0000 7f8349528640 1 -- 172.21.15.16:0/3144984347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83440cbcb0 con 0x7f83440b75c0 2024-04-24T23:49:53.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.231+0000 7f833b7fe640 1 -- 172.21.15.16:0/3144984347 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334037040 con 0x7f83440b75c0 2024-04-24T23:49:53.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.231+0000 7f8349528640 1 -- 172.21.15.16:0/3144984347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83440d09b0 con 0x7f83440b75c0 2024-04-24T23:49:53.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.232+0000 7f8349528640 1 -- 172.21.15.16:0/3144984347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8344068f90 con 0x7f83440b75c0 2024-04-24T23:49:53.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.232+0000 7f833b7fe640 1 -- 172.21.15.16:0/3144984347 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8334044020 con 0x7f83440b75c0 2024-04-24T23:49:53.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.233+0000 7f833b7fe640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f832406be30 0x7f832406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.233+0000 7f833b7fe640 1 -- 172.21.15.16:0/3144984347 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f83340ab9c0 con 0x7f83440b75c0 2024-04-24T23:49:53.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.233+0000 7f833bfff640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f832406be30 0x7f832406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.234+0000 7f833bfff640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f832406be30 0x7f832406e2f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f83441f4c30 tx=0x7f832c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:53.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.237+0000 7f833b7fe640 1 -- 172.21.15.16:0/3144984347 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f83340ade20 con 0x7f83440b75c0 2024-04-24T23:49:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.418+0000 7f8349528640 1 -- 172.21.15.16:0/3144984347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f83440b9210 con 0x7f83440b75c0 2024-04-24T23:49:53.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.418+0000 7f833b7fe640 1 -- 172.21.15.16:0/3144984347 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f833404b030 con 0x7f83440b75c0 2024-04-24T23:49:53.418 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:53.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.421+0000 7f8349528640 1 -- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f832406be30 msgr2=0x7f832406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.421+0000 7f8349528640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f832406be30 0x7f832406e2f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f83441f4c30 tx=0x7f832c009040 comp rx=0 tx=0).stop 2024-04-24T23:49:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.421+0000 7f8349528640 1 -- 172.21.15.16:0/3144984347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83440b75c0 msgr2=0x7f83440d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.421+0000 7f8349528640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83440b75c0 0x7f83440d0470 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f83340090b0 tx=0x7f83340046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.421+0000 7f8349528640 1 -- 172.21.15.16:0/3144984347 shutdown_connections 2024-04-24T23:49:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.421+0000 7f8349528640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f832406be30 0x7f832406e2f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.421+0000 7f8349528640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f834415e2f0 0x7f83440cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.421+0000 7f8349528640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f83440d26e0 0x7f83440cae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.421+0000 7f8349528640 1 --2- 172.21.15.16:0/3144984347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83440b75c0 0x7f83440d0470 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.421+0000 7f8349528640 1 -- 172.21.15.16:0/3144984347 >> 172.21.15.16:0/3144984347 conn(0x7f83440d4850 msgr2=0x7f83440b5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.421+0000 7f8349528640 1 -- 172.21.15.16:0/3144984347 shutdown_connections 2024-04-24T23:49:53.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.421+0000 7f8349528640 1 -- 172.21.15.16:0/3144984347 wait complete. 2024-04-24T23:49:53.431 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:53.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.535+0000 7f29c6a12640 1 Processor -- start 2024-04-24T23:49:53.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.535+0000 7f29c6a12640 1 -- start start 2024-04-24T23:49:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.536+0000 7f29c6a12640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f29c01455f0 0x7f29c006b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.536+0000 7f29c6a12640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f29c00cee70 0x7f29c006ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.536+0000 7f29c6a12640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29c015f9c0 con 0x7f29c006b620 2024-04-24T23:49:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.536+0000 7f29c6a12640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f29c01605d0 con 0x7f29c00cee70 2024-04-24T23:49:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.536+0000 7f29c6a12640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f29c015fca0 con 0x7f29c01455f0 2024-04-24T23:49:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.536+0000 7f29c4faf640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f29c00cee70 0x7f29c006ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.536+0000 7f29bf7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f29c006b620 0x7f29c00ce760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42342/0 (socket says 172.21.15.16:42342) 2024-04-24T23:49:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.536+0000 7f29bf7fe640 1 -- 172.21.15.16:0/4069514394 learned_addr learned my addr 172.21.15.16:0/4069514394 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.536+0000 7f29bffff640 1 --2- 172.21.15.16:0/4069514394 >> v2:172.21.15.156:3300/0 conn(0x7f29c01455f0 0x7f29c006b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.536+0000 7f29c4faf640 1 -- 172.21.15.16:0/4069514394 >> v2:172.21.15.156:3300/0 conn(0x7f29c01455f0 msgr2=0x7f29c006b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.536+0000 7f29c4faf640 1 --2- 172.21.15.16:0/4069514394 >> v2:172.21.15.156:3300/0 conn(0x7f29c01455f0 0x7f29c006b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.536+0000 7f29c4faf640 1 -- 172.21.15.16:0/4069514394 >> v1:172.21.15.16:6789/0 conn(0x7f29c006b620 legacy=0x7f29c00ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.536+0000 7f29beffd640 1 -- 172.21.15.16:0/4069514394 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2556243827 0 0) 0x7f29c015f9c0 con 0x7f29c006b620 2024-04-24T23:49:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.537+0000 7f29c4faf640 1 -- 172.21.15.16:0/4069514394 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c015ff80 con 0x7f29c00cee70 2024-04-24T23:49:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.537+0000 7f29c4faf640 1 --2- 172.21.15.16:0/4069514394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29c00cee70 0x7f29c006ccc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f29c00cf820 tx=0x7f29b802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea46e7e9012f06cc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.537+0000 7f29beffd640 1 -- 172.21.15.16:0/4069514394 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b8007740 con 0x7f29c00cee70 2024-04-24T23:49:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.537+0000 7f29beffd640 1 -- 172.21.15.16:0/4069514394 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29b80078e0 con 0x7f29c00cee70 2024-04-24T23:49:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.537+0000 7f29beffd640 1 -- 172.21.15.16:0/4069514394 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b8007cb0 con 0x7f29c00cee70 2024-04-24T23:49:53.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.538+0000 7f29c6a12640 1 -- 172.21.15.16:0/4069514394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29c00cee70 msgr2=0x7f29c006ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.538+0000 7f29c6a12640 1 --2- 172.21.15.16:0/4069514394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29c00cee70 0x7f29c006ccc0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f29c00cf820 tx=0x7f29b802a5f0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.538+0000 7f29c6a12640 1 -- 172.21.15.16:0/4069514394 shutdown_connections 2024-04-24T23:49:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.538+0000 7f29c6a12640 1 --2- 172.21.15.16:0/4069514394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29c00cee70 0x7f29c006ccc0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.538+0000 7f29c6a12640 1 --2- 172.21.15.16:0/4069514394 >> v2:172.21.15.156:3300/0 conn(0x7f29c01455f0 0x7f29c006b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.538+0000 7f29c6a12640 1 -- 172.21.15.16:0/4069514394 >> 172.21.15.16:0/4069514394 conn(0x7f29c00cfc50 msgr2=0x7f29c00d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.538+0000 7f29c6a12640 1 -- 172.21.15.16:0/4069514394 shutdown_connections 2024-04-24T23:49:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.538+0000 7f29c6a12640 1 -- 172.21.15.16:0/4069514394 wait complete. 2024-04-24T23:49:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.539+0000 7f29c6a12640 1 Processor -- start 2024-04-24T23:49:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.539+0000 7f29c6a12640 1 -- start start 2024-04-24T23:49:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29c6a12640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29c006b620 0x7f29c015f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29c6a12640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29c00cee70 0x7f29c00aeab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29c6a12640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29c01455f0 0x7f29c00c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29c6a12640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f29c015d0a0 con 0x7f29c01455f0 2024-04-24T23:49:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29c6a12640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f29c01459d0 con 0x7f29c00cee70 2024-04-24T23:49:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29c6a12640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f29c015d220 con 0x7f29c006b620 2024-04-24T23:49:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29c4faf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29c01455f0 0x7f29c00c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29c4faf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29c01455f0 0x7f29c00c66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40802/0 (socket says 172.21.15.16:40802) 2024-04-24T23:49:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29c4faf640 1 -- 172.21.15.16:0/3840627462 learned_addr learned my addr 172.21.15.16:0/3840627462 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29bf7fe640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29c00cee70 0x7f29c00aeab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29bffff640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29c006b620 0x7f29c015f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29c4faf640 1 -- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29c006b620 msgr2=0x7f29c015f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29c4faf640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29c006b620 0x7f29c015f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29c4faf640 1 -- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29c00cee70 msgr2=0x7f29c00aeab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29c4faf640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29c00cee70 0x7f29c00aeab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29c4faf640 1 -- 172.21.15.16:0/3840627462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c00ce280 con 0x7f29c01455f0 2024-04-24T23:49:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.540+0000 7f29bf7fe640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29c00cee70 0x7f29c00aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.541+0000 7f29c4faf640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29c01455f0 0x7f29c00c66b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f29b802abe0 tx=0x7f29b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.541+0000 7f29bcff9640 1 -- 172.21.15.16:0/3840627462 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b8040070 con 0x7f29c01455f0 2024-04-24T23:49:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.541+0000 7f29bcff9640 1 -- 172.21.15.16:0/3840627462 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29b8031bc0 con 0x7f29c01455f0 2024-04-24T23:49:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.541+0000 7f29c6a12640 1 -- 172.21.15.16:0/3840627462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c00c4da0 con 0x7f29c01455f0 2024-04-24T23:49:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.541+0000 7f29bcff9640 1 -- 172.21.15.16:0/3840627462 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29b8045460 con 0x7f29c01455f0 2024-04-24T23:49:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.541+0000 7f29c6a12640 1 -- 172.21.15.16:0/3840627462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c015e9f0 con 0x7f29c01455f0 2024-04-24T23:49:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.542+0000 7f29c6a12640 1 -- 172.21.15.16:0/3840627462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c0068f90 con 0x7f29c01455f0 2024-04-24T23:49:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.542+0000 7f29bcff9640 1 -- 172.21.15.16:0/3840627462 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f29b8039040 con 0x7f29c01455f0 2024-04-24T23:49:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.543+0000 7f29bcff9640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f29ac06be60 0x7f29ac06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.543+0000 7f29bcff9640 1 -- 172.21.15.16:0/3840627462 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f29b80acc20 con 0x7f29c01455f0 2024-04-24T23:49:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.543+0000 7f29bffff640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f29ac06be60 0x7f29ac06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.543+0000 7f29bffff640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f29ac06be60 0x7f29ac06e320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f29b4006fd0 tx=0x7f29b4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.547+0000 7f29bcff9640 1 -- 172.21.15.16:0/3840627462 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f29b807b1d0 con 0x7f29c01455f0 2024-04-24T23:49:53.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.730+0000 7f29c6a12640 1 -- 172.21.15.16:0/3840627462 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f29c00b3150 con 0x7f29c01455f0 2024-04-24T23:49:53.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.730+0000 7f29bcff9640 1 -- 172.21.15.16:0/3840627462 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f29b807aad0 con 0x7f29c01455f0 2024-04-24T23:49:53.730 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:53.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.732+0000 7f29c6a12640 1 -- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f29ac06be60 msgr2=0x7f29ac06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.732+0000 7f29c6a12640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f29ac06be60 0x7f29ac06e320 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f29b4006fd0 tx=0x7f29b4008040 comp rx=0 tx=0).stop 2024-04-24T23:49:53.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.732+0000 7f29c6a12640 1 -- 172.21.15.16:0/3840627462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29c01455f0 msgr2=0x7f29c00c66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.732+0000 7f29c6a12640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29c01455f0 0x7f29c00c66b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f29b802abe0 tx=0x7f29b8004a40 comp rx=0 tx=0).stop 2024-04-24T23:49:53.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.732+0000 7f29c6a12640 1 -- 172.21.15.16:0/3840627462 shutdown_connections 2024-04-24T23:49:53.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.732+0000 7f29c6a12640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f29ac06be60 0x7f29ac06e320 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.732+0000 7f29c6a12640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29c01455f0 0x7f29c00c66b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.732+0000 7f29c6a12640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f29c00cee70 0x7f29c00aeab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.732+0000 7f29c6a12640 1 --2- 172.21.15.16:0/3840627462 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f29c006b620 0x7f29c015f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.732+0000 7f29c6a12640 1 -- 172.21.15.16:0/3840627462 >> 172.21.15.16:0/3840627462 conn(0x7f29c00cfc50 msgr2=0x7f29c0143210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:53.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.732+0000 7f29c6a12640 1 -- 172.21.15.16:0/3840627462 shutdown_connections 2024-04-24T23:49:53.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.732+0000 7f29c6a12640 1 -- 172.21.15.16:0/3840627462 wait complete. 2024-04-24T23:49:53.743 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:49:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.845+0000 7ff8a5ba9640 1 Processor -- start 2024-04-24T23:49:53.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.845+0000 7ff8a5ba9640 1 -- start start 2024-04-24T23:49:53.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.846+0000 7ff8a5ba9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8a00ce7c0 0x7ff8a00cec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.846+0000 7ff8a5ba9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8a00cf3a0 con 0x7ff8a0144740 2024-04-24T23:49:53.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.846+0000 7ff8a5ba9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8a00cc8d0 con 0x7ff8a0071a00 2024-04-24T23:49:53.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.846+0000 7ff8a5ba9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff8a015c1f0 con 0x7ff8a00ce7c0 2024-04-24T23:49:53.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.846+0000 7ff89effd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff8a0144740 0x7ff8a015c3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42344/0 (socket says 172.21.15.16:42344) 2024-04-24T23:49:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.846+0000 7ff89effd640 1 -- 172.21.15.16:0/1332924286 learned_addr learned my addr 172.21.15.16:0/1332924286 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.846+0000 7ff89e7fc640 1 -- 172.21.15.16:0/1332924286 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1244429821 0 0) 0x7ff8a00cf3a0 con 0x7ff8a0144740 2024-04-24T23:49:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.846+0000 7ff89ffff640 1 --2- 172.21.15.16:0/1332924286 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8a00ce7c0 0x7ff8a00cec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.846+0000 7ff89e7fc640 1 -- 172.21.15.16:0/1332924286 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff888003520 con 0x7ff8a0144740 2024-04-24T23:49:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.846+0000 7ff89e7fc640 1 -- 172.21.15.16:0/1332924286 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 569254269 0 0) 0x7ff888003520 con 0x7ff8a0144740 2024-04-24T23:49:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.846+0000 7ff89e7fc640 1 -- 172.21.15.16:0/1332924286 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8a00ce7c0 msgr2=0x7ff8a00cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.846+0000 7ff89e7fc640 1 --2- 172.21.15.16:0/1332924286 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8a00ce7c0 0x7ff8a00cec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89e7fc640 1 -- 172.21.15.16:0/1332924286 >> v1:172.21.15.156:6789/0 conn(0x7ff8a0071a00 legacy=0x7ff8a0071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89e7fc640 1 -- 172.21.15.16:0/1332924286 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a00ccbb0 con 0x7ff8a0144740 2024-04-24T23:49:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89e7fc640 1 -- 172.21.15.16:0/1332924286 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7ff894001670 con 0x7ff8a0144740 2024-04-24T23:49:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89e7fc640 1 -- 172.21.15.16:0/1332924286 >> v1:172.21.15.16:6789/0 conn(0x7ff8a0144740 legacy=0x7ff8a015c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89e7fc640 1 --2- 172.21.15.16:0/1332924286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8880039e0 0x7ff888003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89e7fc640 1 --2- 172.21.15.16:0/1332924286 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8a00ce7c0 0x7ff888004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89e7fc640 1 --2- 172.21.15.16:0/1332924286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff888004d40 0x7ff8880051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89e7fc640 1 -- 172.21.15.16:0/1332924286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff88800ba70 con 0x7ff888004d40 2024-04-24T23:49:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89e7fc640 1 -- 172.21.15.16:0/1332924286 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff88800bbf0 con 0x7ff8880039e0 2024-04-24T23:49:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89e7fc640 1 -- 172.21.15.16:0/1332924286 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff88800e000 con 0x7ff8a00ce7c0 2024-04-24T23:49:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89ffff640 1 --2- 172.21.15.16:0/1332924286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff888004d40 0x7ff8880051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89f7fe640 1 --2- 172.21.15.16:0/1332924286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8880039e0 0x7ff888003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89effd640 1 --2- 172.21.15.16:0/1332924286 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8a00ce7c0 0x7ff888004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89ffff640 1 -- 172.21.15.16:0/1332924286 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8a00ce7c0 msgr2=0x7ff888004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89ffff640 1 --2- 172.21.15.16:0/1332924286 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8a00ce7c0 0x7ff888004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89ffff640 1 -- 172.21.15.16:0/1332924286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8880039e0 msgr2=0x7ff888003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89ffff640 1 --2- 172.21.15.16:0/1332924286 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8880039e0 0x7ff888003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89ffff640 1 -- 172.21.15.16:0/1332924286 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff888003520 con 0x7ff888004d40 2024-04-24T23:49:53.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.847+0000 7ff89effd640 1 --2- 172.21.15.16:0/1332924286 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8a00ce7c0 0x7ff888004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:53.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.848+0000 7ff89ffff640 1 --2- 172.21.15.16:0/1332924286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff888004d40 0x7ff8880051c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff898009f90 tx=0x7ff888005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6123533427cb54ea server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:53.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.848+0000 7ff89e7fc640 1 -- 172.21.15.16:0/1332924286 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898009c10 con 0x7ff888004d40 2024-04-24T23:49:53.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.848+0000 7ff89e7fc640 1 -- 172.21.15.16:0/1332924286 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff898036070 con 0x7ff888004d40 2024-04-24T23:49:53.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.848+0000 7ff89e7fc640 1 -- 172.21.15.16:0/1332924286 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898031850 con 0x7ff888004d40 2024-04-24T23:49:53.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.848+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/1332924286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff888004d40 msgr2=0x7ff8880051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.848+0000 7ff8a5ba9640 1 --2- 172.21.15.16:0/1332924286 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff888004d40 0x7ff8880051c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff898009f90 tx=0x7ff888005830 comp rx=0 tx=0).stop 2024-04-24T23:49:53.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.848+0000 7ff89ffff640 1 -- 172.21.15.16:0/1332924286 reap_dead start 2024-04-24T23:49:53.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.848+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/1332924286 shutdown_connections 2024-04-24T23:49:53.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.848+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/1332924286 >> 172.21.15.16:0/1332924286 conn(0x7ff8a00b2000 msgr2=0x7ff8a00cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:53.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.848+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/1332924286 shutdown_connections 2024-04-24T23:49:53.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.848+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/1332924286 wait complete. 2024-04-24T23:49:53.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.850+0000 7ff8a5ba9640 1 Processor -- start 2024-04-24T23:49:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.850+0000 7ff8a5ba9640 1 -- start start 2024-04-24T23:49:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.850+0000 7ff8a5ba9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8a0150220 0x7ff8a0141f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.850+0000 7ff8a5ba9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8a01507e0 0x7ff8a0143810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff8a5ba9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8a015b950 0x7ff8a015be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff8a5ba9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a015cab0 con 0x7ff8a015b950 2024-04-24T23:49:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff8a5ba9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a00cd170 con 0x7ff8a0150220 2024-04-24T23:49:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff8a5ba9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff8a00cec50 con 0x7ff8a01507e0 2024-04-24T23:49:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff89ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8a015b950 0x7ff8a015be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff89ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8a015b950 0x7ff8a015be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40818/0 (socket says 172.21.15.16:40818) 2024-04-24T23:49:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff89ffff640 1 -- 172.21.15.16:0/2256968130 learned_addr learned my addr 172.21.15.16:0/2256968130 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff89effd640 1 --2- 172.21.15.16:0/2256968130 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8a01507e0 0x7ff8a0143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff89f7fe640 1 --2- 172.21.15.16:0/2256968130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8a0150220 0x7ff8a0141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff89ffff640 1 -- 172.21.15.16:0/2256968130 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8a01507e0 msgr2=0x7ff8a0143810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff89ffff640 1 --2- 172.21.15.16:0/2256968130 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8a01507e0 0x7ff8a0143810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff89ffff640 1 -- 172.21.15.16:0/2256968130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8a0150220 msgr2=0x7ff8a0141f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:53.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff89ffff640 1 --2- 172.21.15.16:0/2256968130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8a0150220 0x7ff8a0141f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:53.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff89ffff640 1 -- 172.21.15.16:0/2256968130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a0142770 con 0x7ff8a015b950 2024-04-24T23:49:53.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff89ffff640 1 --2- 172.21.15.16:0/2256968130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8a015b950 0x7ff8a015be00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff898033040 tx=0x7ff898009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:53.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff887fff640 1 -- 172.21.15.16:0/2256968130 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898009c10 con 0x7ff8a015b950 2024-04-24T23:49:53.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff887fff640 1 -- 172.21.15.16:0/2256968130 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff898003990 con 0x7ff8a015b950 2024-04-24T23:49:53.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/2256968130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a0142a50 con 0x7ff8a015b950 2024-04-24T23:49:53.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff887fff640 1 -- 172.21.15.16:0/2256968130 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898003d80 con 0x7ff8a015b950 2024-04-24T23:49:53.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.851+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/2256968130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a0143010 con 0x7ff8a015b950 2024-04-24T23:49:53.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.852+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/2256968130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8a00731f0 con 0x7ff8a015b950 2024-04-24T23:49:53.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.853+0000 7ff887fff640 1 -- 172.21.15.16:0/2256968130 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff898005d10 con 0x7ff8a015b950 2024-04-24T23:49:53.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.853+0000 7ff887fff640 1 --2- 172.21.15.16:0/2256968130 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff88c06be60 0x7ff88c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:53.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.853+0000 7ff887fff640 1 -- 172.21.15.16:0/2256968130 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ff8980ac510 con 0x7ff8a015b950 2024-04-24T23:49:53.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.854+0000 7ff89f7fe640 1 --2- 172.21.15.16:0/2256968130 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff88c06be60 0x7ff88c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:53.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.854+0000 7ff89f7fe640 1 --2- 172.21.15.16:0/2256968130 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff88c06be60 0x7ff88c06e320 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff890001760 tx=0x7ff890009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:53.858 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:53.857+0000 7ff887fff640 1 -- 172.21.15.16:0/2256968130 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff89804a060 con 0x7ff8a015b950 2024-04-24T23:49:53.996 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:49:54.000 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:49:54.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.055+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/2256968130 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff8a015c890 con 0x7ff8a015b950 2024-04-24T23:49:54.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.056+0000 7ff887fff640 1 -- 172.21.15.16:0/2256968130 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7ff89802e050 con 0x7ff8a015b950 2024-04-24T23:49:54.058 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-04-24T23:49:54.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.058+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/2256968130 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff88c06be60 msgr2=0x7ff88c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.058+0000 7ff8a5ba9640 1 --2- 172.21.15.16:0/2256968130 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff88c06be60 0x7ff88c06e320 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff890001760 tx=0x7ff890009040 comp rx=0 tx=0).stop 2024-04-24T23:49:54.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.058+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/2256968130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8a015b950 msgr2=0x7ff8a015be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.058+0000 7ff8a5ba9640 1 --2- 172.21.15.16:0/2256968130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8a015b950 0x7ff8a015be00 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff898033040 tx=0x7ff898009750 comp rx=0 tx=0).stop 2024-04-24T23:49:54.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.058+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/2256968130 shutdown_connections 2024-04-24T23:49:54.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.058+0000 7ff8a5ba9640 1 --2- 172.21.15.16:0/2256968130 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff88c06be60 0x7ff88c06e320 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.058+0000 7ff8a5ba9640 1 --2- 172.21.15.16:0/2256968130 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8a015b950 0x7ff8a015be00 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.058+0000 7ff8a5ba9640 1 --2- 172.21.15.16:0/2256968130 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff8a01507e0 0x7ff8a0143810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.058+0000 7ff8a5ba9640 1 --2- 172.21.15.16:0/2256968130 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff8a0150220 0x7ff8a0141f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.058+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/2256968130 >> 172.21.15.16:0/2256968130 conn(0x7ff8a00b2000 msgr2=0x7ff8a0140860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:54.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.058+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/2256968130 shutdown_connections 2024-04-24T23:49:54.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.058+0000 7ff8a5ba9640 1 -- 172.21.15.16:0/2256968130 wait complete. 2024-04-24T23:49:54.068 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:49:54.069 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:49:54.087 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.210+0000 7fa70ea6b640 1 Processor -- start 2024-04-24T23:49:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.210+0000 7fa70ea6b640 1 -- start start 2024-04-24T23:49:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.211+0000 7fa70ea6b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa708071a00 0x7fa708071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.211+0000 7fa70ea6b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa7080ce7c0 0x7fa7080cebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.211+0000 7fa70ea6b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7080cf130 con 0x7fa708144740 2024-04-24T23:49:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.211+0000 7fa70ea6b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa70815c1f0 con 0x7fa708071a00 2024-04-24T23:49:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.211+0000 7fa70ea6b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa708072320 con 0x7fa7080ce7c0 2024-04-24T23:49:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.211+0000 7fa70c807640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa708071a00 0x7fa708071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.211+0000 7fa70c807640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa708071a00 0x7fa708071de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40834/0 (socket says 172.21.15.16:40834) 2024-04-24T23:49:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.211+0000 7fa70c807640 1 -- 172.21.15.16:0/914826151 learned_addr learned my addr 172.21.15.16:0/914826151 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.211+0000 7fa70d008640 1 --2- 172.21.15.16:0/914826151 >> v2:172.21.15.156:3300/0 conn(0x7fa7080ce7c0 0x7fa7080cebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.211+0000 7fa70c807640 1 -- 172.21.15.16:0/914826151 >> v2:172.21.15.156:3300/0 conn(0x7fa7080ce7c0 msgr2=0x7fa7080cebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.211+0000 7fa70c807640 1 --2- 172.21.15.16:0/914826151 >> v2:172.21.15.156:3300/0 conn(0x7fa7080ce7c0 0x7fa7080cebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.211+0000 7fa70c807640 1 -- 172.21.15.16:0/914826151 >> v1:172.21.15.156:6789/0 conn(0x7fa708144740 legacy=0x7fa70815c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.211+0000 7fa70c807640 1 -- 172.21.15.16:0/914826151 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7080cf3a0 con 0x7fa708071a00 2024-04-24T23:49:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.212+0000 7fa70c807640 1 --2- 172.21.15.16:0/914826151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa708071a00 0x7fa708071de0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa7080726e0 tx=0x7fa6fc02a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525d41d607a6e4f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.212+0000 7fa7077fe640 1 -- 172.21.15.16:0/914826151 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc037070 con 0x7fa708071a00 2024-04-24T23:49:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.212+0000 7fa7077fe640 1 -- 172.21.15.16:0/914826151 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6fc032440 con 0x7fa708071a00 2024-04-24T23:49:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.212+0000 7fa7077fe640 1 -- 172.21.15.16:0/914826151 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc032810 con 0x7fa708071a00 2024-04-24T23:49:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.213+0000 7fa70ea6b640 1 -- 172.21.15.16:0/914826151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa708071a00 msgr2=0x7fa708071de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.213+0000 7fa70ea6b640 1 --2- 172.21.15.16:0/914826151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa708071a00 0x7fa708071de0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa7080726e0 tx=0x7fa6fc02a230 comp rx=0 tx=0).stop 2024-04-24T23:49:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.213+0000 7fa70ea6b640 1 -- 172.21.15.16:0/914826151 shutdown_connections 2024-04-24T23:49:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.213+0000 7fa70ea6b640 1 --2- 172.21.15.16:0/914826151 >> v2:172.21.15.156:3300/0 conn(0x7fa7080ce7c0 0x7fa7080cebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.213+0000 7fa70ea6b640 1 --2- 172.21.15.16:0/914826151 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa708071a00 0x7fa708071de0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.213+0000 7fa70ea6b640 1 -- 172.21.15.16:0/914826151 >> 172.21.15.16:0/914826151 conn(0x7fa7080b2000 msgr2=0x7fa7080cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:54.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.213+0000 7fa70ea6b640 1 -- 172.21.15.16:0/914826151 shutdown_connections 2024-04-24T23:49:54.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.213+0000 7fa70ea6b640 1 -- 172.21.15.16:0/914826151 wait complete. 2024-04-24T23:49:54.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.214+0000 7fa70ea6b640 1 Processor -- start 2024-04-24T23:49:54.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.214+0000 7fa70ea6b640 1 -- start start 2024-04-24T23:49:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70ea6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa708071a00 0x7fa70813a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70ea6b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7080ce7c0 0x7fa7081e3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70ea6b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7081e39b0 0x7fa7081428a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70ea6b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa7080cce90 con 0x7fa708071a00 2024-04-24T23:49:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70ea6b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa7080ccbb0 con 0x7fa7080ce7c0 2024-04-24T23:49:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70ea6b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fa7080cd170 con 0x7fa7081e39b0 2024-04-24T23:49:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70c807640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa708071a00 0x7fa70813a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70c807640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa708071a00 0x7fa70813a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40838/0 (socket says 172.21.15.16:40838) 2024-04-24T23:49:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70c807640 1 -- 172.21.15.16:0/145140862 learned_addr learned my addr 172.21.15.16:0/145140862 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa707fff640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7080ce7c0 0x7fa7081e3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70d008640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7081e39b0 0x7fa7081428a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70c807640 1 -- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7081e39b0 msgr2=0x7fa7081428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70c807640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7081e39b0 0x7fa7081428a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70c807640 1 -- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7080ce7c0 msgr2=0x7fa7081e3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70c807640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7080ce7c0 0x7fa7081e3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70c807640 1 -- 172.21.15.16:0/145140862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7081e3ec0 con 0x7fa708071a00 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70d008640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7081e39b0 0x7fa7081428a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa707fff640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7080ce7c0 0x7fa7081e3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.215+0000 7fa70c807640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa708071a00 0x7fa70813a530 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc000c00 tx=0x7fa6fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.216+0000 7fa7057fa640 1 -- 172.21.15.16:0/145140862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc041020 con 0x7fa708071a00 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.216+0000 7fa7057fa640 1 -- 172.21.15.16:0/145140862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6fc0306c0 con 0x7fa708071a00 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.216+0000 7fa70ea6b640 1 -- 172.21.15.16:0/145140862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7081e41a0 con 0x7fa708071a00 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.216+0000 7fa7057fa640 1 -- 172.21.15.16:0/145140862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc037040 con 0x7fa708071a00 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.216+0000 7fa70ea6b640 1 -- 172.21.15.16:0/145140862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7081e4760 con 0x7fa708071a00 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.217+0000 7fa70ea6b640 1 -- 172.21.15.16:0/145140862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7080731f0 con 0x7fa708071a00 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.217+0000 7fa7057fa640 1 -- 172.21.15.16:0/145140862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa6fc044020 con 0x7fa708071a00 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.218+0000 7fa7057fa640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa6ec06be30 0x7fa6ec06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.218+0000 7fa7057fa640 1 -- 172.21.15.16:0/145140862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa6fc0aba10 con 0x7fa708071a00 2024-04-24T23:49:54.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.218+0000 7fa707fff640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa6ec06be30 0x7fa6ec06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.218+0000 7fa707fff640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa6ec06be30 0x7fa6ec06e2f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa70815ad60 tx=0x7fa6f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:54.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.222+0000 7fa7057fa640 1 -- 172.21.15.16:0/145140862 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa6fc04b030 con 0x7fa708071a00 2024-04-24T23:49:54.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.407+0000 7fa70ea6b640 1 -- 172.21.15.16:0/145140862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa70804f5e0 con 0x7fa708071a00 2024-04-24T23:49:54.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.408+0000 7fa7057fa640 1 -- 172.21.15.16:0/145140862 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7fa6fc07a050 con 0x7fa708071a00 2024-04-24T23:49:54.408 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:54.408 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:54.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.410+0000 7fa70ea6b640 1 -- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa6ec06be30 msgr2=0x7fa6ec06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.410+0000 7fa70ea6b640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa6ec06be30 0x7fa6ec06e2f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa70815ad60 tx=0x7fa6f4008040 comp rx=0 tx=0).stop 2024-04-24T23:49:54.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.410+0000 7fa70ea6b640 1 -- 172.21.15.16:0/145140862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa708071a00 msgr2=0x7fa70813a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.410+0000 7fa70ea6b640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa708071a00 0x7fa70813a530 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc000c00 tx=0x7fa6fc0046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.410+0000 7fa70ea6b640 1 -- 172.21.15.16:0/145140862 shutdown_connections 2024-04-24T23:49:54.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.410+0000 7fa70ea6b640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa6ec06be30 0x7fa6ec06e2f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.410+0000 7fa70ea6b640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa7081e39b0 0x7fa7081428a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.410+0000 7fa70ea6b640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa7080ce7c0 0x7fa7081e3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.410+0000 7fa70ea6b640 1 --2- 172.21.15.16:0/145140862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa708071a00 0x7fa70813a530 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.410+0000 7fa70ea6b640 1 -- 172.21.15.16:0/145140862 >> 172.21.15.16:0/145140862 conn(0x7fa7080b2000 msgr2=0x7fa7080afc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:54.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.410+0000 7fa70ea6b640 1 -- 172.21.15.16:0/145140862 shutdown_connections 2024-04-24T23:49:54.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.410+0000 7fa70ea6b640 1 -- 172.21.15.16:0/145140862 wait complete. 2024-04-24T23:49:54.421 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:54.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:54.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.525+0000 7f0a8ae6b640 1 Processor -- start 2024-04-24T23:49:54.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.525+0000 7f0a8ae6b640 1 -- start start 2024-04-24T23:49:54.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a8ae6b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a840d2130 0x7f0a840d2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a8ae6b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0a840c80a0 0x7f0a840c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a8ae6b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a840d1390 con 0x7f0a840c7cc0 2024-04-24T23:49:54.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a8ae6b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f0a840d2ad0 con 0x7f0a840c80a0 2024-04-24T23:49:54.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a8ae6b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0a84160940 con 0x7f0a840d2130 2024-04-24T23:49:54.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a89408640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0a840c80a0 0x7f0a840c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a88c07640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0a840c7cc0 0x7f0a84160230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42348/0 (socket says 172.21.15.16:42348) 2024-04-24T23:49:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a89408640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0a840c80a0 0x7f0a840c84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40846/0 (socket says 172.21.15.16:40846) 2024-04-24T23:49:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a88c07640 1 -- 172.21.15.16:0/3169915983 learned_addr learned my addr 172.21.15.16:0/3169915983 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a7bfff640 1 --2- 172.21.15.16:0/3169915983 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a840d2130 0x7f0a840d2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a89408640 1 -- 172.21.15.16:0/3169915983 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a840d2130 msgr2=0x7f0a840d2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a89408640 1 --2- 172.21.15.16:0/3169915983 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a840d2130 0x7f0a840d2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a89408640 1 -- 172.21.15.16:0/3169915983 >> v1:172.21.15.16:6789/0 conn(0x7f0a840c7cc0 legacy=0x7f0a84160230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a89408640 1 -- 172.21.15.16:0/3169915983 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a840d1670 con 0x7f0a840c80a0 2024-04-24T23:49:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.526+0000 7f0a7b7fe640 1 -- 172.21.15.16:0/3169915983 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3906043214 0 0) 0x7f0a840d1390 con 0x7f0a840c7cc0 2024-04-24T23:49:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.527+0000 7f0a89408640 1 --2- 172.21.15.16:0/3169915983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a840c80a0 0x7f0a840c84f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c009d90 tx=0x7f0a7c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788a49a39ba9b7a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:54.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.527+0000 7f0a7b7fe640 1 -- 172.21.15.16:0/3169915983 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c007740 con 0x7f0a840c80a0 2024-04-24T23:49:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.527+0000 7f0a7b7fe640 1 -- 172.21.15.16:0/3169915983 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a7c0078e0 con 0x7f0a840c80a0 2024-04-24T23:49:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.527+0000 7f0a7b7fe640 1 -- 172.21.15.16:0/3169915983 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7c007cb0 con 0x7f0a840c80a0 2024-04-24T23:49:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.528+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/3169915983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a840c80a0 msgr2=0x7f0a840c84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.528+0000 7f0a8ae6b640 1 --2- 172.21.15.16:0/3169915983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a840c80a0 0x7f0a840c84f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c009d90 tx=0x7f0a7c02a660 comp rx=0 tx=0).stop 2024-04-24T23:49:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.528+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/3169915983 shutdown_connections 2024-04-24T23:49:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.528+0000 7f0a8ae6b640 1 --2- 172.21.15.16:0/3169915983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a840c80a0 0x7f0a840c84f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.528+0000 7f0a8ae6b640 1 --2- 172.21.15.16:0/3169915983 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a840d2130 0x7f0a840d2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.528+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/3169915983 >> 172.21.15.16:0/3169915983 conn(0x7f0a84072e70 msgr2=0x7f0a84073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.528+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/3169915983 shutdown_connections 2024-04-24T23:49:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.528+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/3169915983 wait complete. 2024-04-24T23:49:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.529+0000 7f0a8ae6b640 1 Processor -- start 2024-04-24T23:49:54.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.529+0000 7f0a8ae6b640 1 -- start start 2024-04-24T23:49:54.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a8ae6b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a840c7cc0 0x7f0a8406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a8ae6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a840c80a0 0x7f0a84136d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a8ae6b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a840d2130 0x7f0a840b0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a8ae6b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0a840c8a30 con 0x7f0a840c80a0 2024-04-24T23:49:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a8ae6b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0a8415fd10 con 0x7f0a840c7cc0 2024-04-24T23:49:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a8ae6b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0a840cfaa0 con 0x7f0a840d2130 2024-04-24T23:49:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a7bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a840c80a0 0x7f0a84136d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a7bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a840c80a0 0x7f0a84136d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40860/0 (socket says 172.21.15.16:40860) 2024-04-24T23:49:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a7bfff640 1 -- 172.21.15.16:0/235895174 learned_addr learned my addr 172.21.15.16:0/235895174 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a88c07640 1 --2- 172.21.15.16:0/235895174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a840c7cc0 0x7f0a8406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a89408640 1 --2- 172.21.15.16:0/235895174 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a840d2130 0x7f0a840b0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a7bfff640 1 -- 172.21.15.16:0/235895174 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a840d2130 msgr2=0x7f0a840b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a7bfff640 1 --2- 172.21.15.16:0/235895174 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a840d2130 0x7f0a840b0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a7bfff640 1 -- 172.21.15.16:0/235895174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a840c7cc0 msgr2=0x7f0a8406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.532 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a7bfff640 1 --2- 172.21.15.16:0/235895174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a840c7cc0 0x7f0a8406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.530+0000 7f0a7bfff640 1 -- 172.21.15.16:0/235895174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a8406d120 con 0x7f0a840c80a0 2024-04-24T23:49:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.531+0000 7f0a7bfff640 1 --2- 172.21.15.16:0/235895174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a840c80a0 0x7f0a84136d00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400e8a0 tx=0x7f0a7400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.531+0000 7f0a797fa640 1 -- 172.21.15.16:0/235895174 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a74017070 con 0x7f0a840c80a0 2024-04-24T23:49:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.531+0000 7f0a797fa640 1 -- 172.21.15.16:0/235895174 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a7400c940 con 0x7f0a840c80a0 2024-04-24T23:49:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.531+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/235895174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a8406d400 con 0x7f0a840c80a0 2024-04-24T23:49:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.531+0000 7f0a797fa640 1 -- 172.21.15.16:0/235895174 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7400cd30 con 0x7f0a840c80a0 2024-04-24T23:49:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.531+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/235895174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a8406d9c0 con 0x7f0a840c80a0 2024-04-24T23:49:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.532+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/235895174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a84068f90 con 0x7f0a840c80a0 2024-04-24T23:49:54.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.532+0000 7f0a797fa640 1 -- 172.21.15.16:0/235895174 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0a7401a070 con 0x7f0a840c80a0 2024-04-24T23:49:54.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.533+0000 7f0a797fa640 1 --2- 172.21.15.16:0/235895174 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0a7006be30 0x7f0a7006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.533+0000 7f0a797fa640 1 -- 172.21.15.16:0/235895174 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0a7408b370 con 0x7f0a840c80a0 2024-04-24T23:49:54.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.533+0000 7f0a88c07640 1 --2- 172.21.15.16:0/235895174 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0a7006be30 0x7f0a7006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.533+0000 7f0a88c07640 1 --2- 172.21.15.16:0/235895174 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0a7006be30 0x7f0a7006e2f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0a80000f60 tx=0x7f0a80007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:54.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.537+0000 7f0a797fa640 1 -- 172.21.15.16:0/235895174 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0a74059360 con 0x7f0a840c80a0 2024-04-24T23:49:54.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.719+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/235895174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0a840c85b0 con 0x7f0a840c80a0 2024-04-24T23:49:54.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.719+0000 7f0a797fa640 1 -- 172.21.15.16:0/235895174 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f0a7405d660 con 0x7f0a840c80a0 2024-04-24T23:49:54.719 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:54.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.721+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/235895174 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0a7006be30 msgr2=0x7f0a7006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.721+0000 7f0a8ae6b640 1 --2- 172.21.15.16:0/235895174 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0a7006be30 0x7f0a7006e2f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0a80000f60 tx=0x7f0a80007d50 comp rx=0 tx=0).stop 2024-04-24T23:49:54.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.721+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/235895174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a840c80a0 msgr2=0x7f0a84136d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.721+0000 7f0a8ae6b640 1 --2- 172.21.15.16:0/235895174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a840c80a0 0x7f0a84136d00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400e8a0 tx=0x7f0a7400ed60 comp rx=0 tx=0).stop 2024-04-24T23:49:54.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.721+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/235895174 shutdown_connections 2024-04-24T23:49:54.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.721+0000 7f0a8ae6b640 1 --2- 172.21.15.16:0/235895174 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0a7006be30 0x7f0a7006e2f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.721+0000 7f0a8ae6b640 1 --2- 172.21.15.16:0/235895174 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0a840d2130 0x7f0a840b0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.721+0000 7f0a8ae6b640 1 --2- 172.21.15.16:0/235895174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a840c80a0 0x7f0a84136d00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.721+0000 7f0a8ae6b640 1 --2- 172.21.15.16:0/235895174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0a840c7cc0 0x7f0a8406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.721+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/235895174 >> 172.21.15.16:0/235895174 conn(0x7f0a84072e70 msgr2=0x7f0a840b17a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:54.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.721+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/235895174 shutdown_connections 2024-04-24T23:49:54.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.721+0000 7f0a8ae6b640 1 -- 172.21.15.16:0/235895174 wait complete. 2024-04-24T23:49:54.732 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.836+0000 7fd55ba7d640 1 Processor -- start 2024-04-24T23:49:54.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.836+0000 7fd55ba7d640 1 -- start start 2024-04-24T23:49:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.837+0000 7fd55ba7d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd5540d12d0 0x7fd5540d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.837+0000 7fd55ba7d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd5540cd790 0x7fd55415e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.837+0000 7fd55ba7d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd55415ec30 con 0x7fd5540c8580 2024-04-24T23:49:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.837+0000 7fd55ba7d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd55415f4d0 con 0x7fd5540cd790 2024-04-24T23:49:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.837+0000 7fd55ba7d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fd55415ef10 con 0x7fd5540d12d0 2024-04-24T23:49:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.837+0000 7fd559018640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd5540cd790 0x7fd55415e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.837+0000 7fd55a01a640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd5540c8580 0x7fd5540d2130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42360/0 (socket says 172.21.15.16:42360) 2024-04-24T23:49:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.837+0000 7fd55a01a640 1 -- 172.21.15.16:0/3637791750 learned_addr learned my addr 172.21.15.16:0/3637791750 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.837+0000 7fd559819640 1 --2- 172.21.15.16:0/3637791750 >> v2:172.21.15.156:3300/0 conn(0x7fd5540d12d0 0x7fd5540d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.837+0000 7fd559018640 1 -- 172.21.15.16:0/3637791750 >> v2:172.21.15.156:3300/0 conn(0x7fd5540d12d0 msgr2=0x7fd5540d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:54.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.837+0000 7fd559018640 1 --2- 172.21.15.16:0/3637791750 >> v2:172.21.15.156:3300/0 conn(0x7fd5540d12d0 0x7fd5540d16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.837+0000 7fd559018640 1 -- 172.21.15.16:0/3637791750 >> v1:172.21.15.16:6789/0 conn(0x7fd5540c8580 legacy=0x7fd5540d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.837+0000 7fd559018640 1 -- 172.21.15.16:0/3637791750 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd55415f1f0 con 0x7fd5540cd790 2024-04-24T23:49:54.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.837+0000 7fd558817640 1 -- 172.21.15.16:0/3637791750 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 309928550 0 0) 0x7fd55415ec30 con 0x7fd5540c8580 2024-04-24T23:49:54.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.838+0000 7fd559018640 1 --2- 172.21.15.16:0/3637791750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5540cd790 0x7fd55415e5c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd5540c7600 tx=0x7fd54402a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2706b6a93769f32e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:54.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.838+0000 7fd558817640 1 -- 172.21.15.16:0/3637791750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd544037070 con 0x7fd5540cd790 2024-04-24T23:49:54.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.838+0000 7fd558817640 1 -- 172.21.15.16:0/3637791750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd54402add0 con 0x7fd5540cd790 2024-04-24T23:49:54.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.838+0000 7fd558817640 1 -- 172.21.15.16:0/3637791750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd544032660 con 0x7fd5540cd790 2024-04-24T23:49:54.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.839+0000 7fd55ba7d640 1 -- 172.21.15.16:0/3637791750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5540cd790 msgr2=0x7fd55415e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.839+0000 7fd55ba7d640 1 --2- 172.21.15.16:0/3637791750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5540cd790 0x7fd55415e5c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd5540c7600 tx=0x7fd54402a080 comp rx=0 tx=0).stop 2024-04-24T23:49:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.839+0000 7fd55ba7d640 1 -- 172.21.15.16:0/3637791750 shutdown_connections 2024-04-24T23:49:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.839+0000 7fd55ba7d640 1 --2- 172.21.15.16:0/3637791750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5540cd790 0x7fd55415e5c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.839+0000 7fd55ba7d640 1 --2- 172.21.15.16:0/3637791750 >> v2:172.21.15.156:3300/0 conn(0x7fd5540d12d0 0x7fd5540d16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.839+0000 7fd55ba7d640 1 -- 172.21.15.16:0/3637791750 >> 172.21.15.16:0/3637791750 conn(0x7fd554072e70 msgr2=0x7fd554073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.839+0000 7fd55ba7d640 1 -- 172.21.15.16:0/3637791750 shutdown_connections 2024-04-24T23:49:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.839+0000 7fd55ba7d640 1 -- 172.21.15.16:0/3637791750 wait complete. 2024-04-24T23:49:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.840+0000 7fd55ba7d640 1 Processor -- start 2024-04-24T23:49:54.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.840+0000 7fd55ba7d640 1 -- start start 2024-04-24T23:49:54.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd55ba7d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5540c8580 0x7fd5540b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd55ba7d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5540cd790 0x7fd5541574d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd55ba7d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5540d12d0 0x7fd554157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd55ba7d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd5540c8b50 con 0x7fd5540c8580 2024-04-24T23:49:54.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd55ba7d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd5540c89d0 con 0x7fd5540cd790 2024-04-24T23:49:54.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd55ba7d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fd5540d2840 con 0x7fd5540d12d0 2024-04-24T23:49:54.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd559819640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5540c8580 0x7fd5540b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd559819640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5540c8580 0x7fd5540b49a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40890/0 (socket says 172.21.15.16:40890) 2024-04-24T23:49:54.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd559819640 1 -- 172.21.15.16:0/2823430932 learned_addr learned my addr 172.21.15.16:0/2823430932 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:54.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd559018640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5540cd790 0x7fd5541574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd55a01a640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5540d12d0 0x7fd554157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd559819640 1 -- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5540d12d0 msgr2=0x7fd554157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd559819640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5540d12d0 0x7fd554157a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd559819640 1 -- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5540cd790 msgr2=0x7fd5541574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:54.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd559819640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5540cd790 0x7fd5541574d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:54.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd559819640 1 -- 172.21.15.16:0/2823430932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5540ca750 con 0x7fd5540c8580 2024-04-24T23:49:54.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd559018640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5540cd790 0x7fd5541574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:54.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd559819640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5540c8580 0x7fd5540b49a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd55000e870 tx=0x7fd55000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:54.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd54a7fc640 1 -- 172.21.15.16:0/2823430932 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd550017070 con 0x7fd5540c8580 2024-04-24T23:49:54.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd54a7fc640 1 -- 172.21.15.16:0/2823430932 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd55000c940 con 0x7fd5540c8580 2024-04-24T23:49:54.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd55ba7d640 1 -- 172.21.15.16:0/2823430932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd554158070 con 0x7fd5540c8580 2024-04-24T23:49:54.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd54a7fc640 1 -- 172.21.15.16:0/2823430932 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd55000cd30 con 0x7fd5540c8580 2024-04-24T23:49:54.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.841+0000 7fd55ba7d640 1 -- 172.21.15.16:0/2823430932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5540d0c20 con 0x7fd5540c8580 2024-04-24T23:49:54.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.842+0000 7fd55ba7d640 1 -- 172.21.15.16:0/2823430932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd554068f90 con 0x7fd5540c8580 2024-04-24T23:49:54.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.843+0000 7fd54a7fc640 1 -- 172.21.15.16:0/2823430932 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd55001a070 con 0x7fd5540c8580 2024-04-24T23:49:54.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.843+0000 7fd54a7fc640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd53806be30 0x7fd53806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:54.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.843+0000 7fd54a7fc640 1 -- 172.21.15.16:0/2823430932 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fd55008b110 con 0x7fd5540c8580 2024-04-24T23:49:54.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.844+0000 7fd559018640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd53806be30 0x7fd53806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:54.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.844+0000 7fd559018640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd53806be30 0x7fd53806e2f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd544034000 tx=0x7fd54402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:54.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:54.848+0000 7fd54a7fc640 1 -- 172.21.15.16:0/2823430932 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd5500596c0 con 0x7fd5540c8580 2024-04-24T23:49:55.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.035+0000 7fd55ba7d640 1 -- 172.21.15.16:0/2823430932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd5540b35a0 con 0x7fd5540c8580 2024-04-24T23:49:55.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.036+0000 7fd54a7fc640 1 -- 172.21.15.16:0/2823430932 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fd550058fc0 con 0x7fd5540c8580 2024-04-24T23:49:55.036 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:55.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.038+0000 7fd55ba7d640 1 -- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd53806be30 msgr2=0x7fd53806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.038+0000 7fd55ba7d640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd53806be30 0x7fd53806e2f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd544034000 tx=0x7fd54402e000 comp rx=0 tx=0).stop 2024-04-24T23:49:55.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.038+0000 7fd55ba7d640 1 -- 172.21.15.16:0/2823430932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5540c8580 msgr2=0x7fd5540b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.038+0000 7fd55ba7d640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5540c8580 0x7fd5540b49a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd55000e870 tx=0x7fd55000ed30 comp rx=0 tx=0).stop 2024-04-24T23:49:55.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.038+0000 7fd55ba7d640 1 -- 172.21.15.16:0/2823430932 shutdown_connections 2024-04-24T23:49:55.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.038+0000 7fd55ba7d640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd53806be30 0x7fd53806e2f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.038+0000 7fd55ba7d640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd5540d12d0 0x7fd554157a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.038+0000 7fd55ba7d640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd5540cd790 0x7fd5541574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.038+0000 7fd55ba7d640 1 --2- 172.21.15.16:0/2823430932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd5540c8580 0x7fd5540b49a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.038+0000 7fd55ba7d640 1 -- 172.21.15.16:0/2823430932 >> 172.21.15.16:0/2823430932 conn(0x7fd554072e70 msgr2=0x7fd554073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:55.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.038+0000 7fd55ba7d640 1 -- 172.21.15.16:0/2823430932 shutdown_connections 2024-04-24T23:49:55.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.038+0000 7fd55ba7d640 1 -- 172.21.15.16:0/2823430932 wait complete. 2024-04-24T23:49:55.049 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:55.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.153+0000 7fbc04c9e640 1 Processor -- start 2024-04-24T23:49:55.153 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.153+0000 7fbc04c9e640 1 -- start start 2024-04-24T23:49:55.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.154+0000 7fbc04c9e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc000cd790 0x7fbc000d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.154+0000 7fbc04c9e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbc0015e5c0 0x7fbc000d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.154+0000 7fbc04c9e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc0015ec30 con 0x7fbc000d12d0 2024-04-24T23:49:55.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.154+0000 7fbc04c9e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fbc0015f4d0 con 0x7fbc000cd790 2024-04-24T23:49:55.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.154+0000 7fbc04c9e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fbc0015ef10 con 0x7fbc0015e5c0 2024-04-24T23:49:55.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.154+0000 7fbbf7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc000cd790 0x7fbc000d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.154+0000 7fbbf7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc000cd790 0x7fbc000d16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40898/0 (socket says 172.21.15.16:40898) 2024-04-24T23:49:55.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.154+0000 7fbbf7fff640 1 -- 172.21.15.16:0/3336326787 learned_addr learned my addr 172.21.15.16:0/3336326787 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:55.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.154+0000 7fbbfed9d640 1 --2- 172.21.15.16:0/3336326787 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbc0015e5c0 0x7fbc000d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.154+0000 7fbbf7fff640 1 -- 172.21.15.16:0/3336326787 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbc0015e5c0 msgr2=0x7fbc000d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:55.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.154+0000 7fbbf7fff640 1 --2- 172.21.15.16:0/3336326787 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbc0015e5c0 0x7fbc000d2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.154+0000 7fbbf7fff640 1 -- 172.21.15.16:0/3336326787 >> v1:172.21.15.156:6789/0 conn(0x7fbc000d12d0 legacy=0x7fbc000c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.154+0000 7fbbf7fff640 1 -- 172.21.15.16:0/3336326787 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0015f1f0 con 0x7fbc000cd790 2024-04-24T23:49:55.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.155+0000 7fbbf7fff640 1 --2- 172.21.15.16:0/3336326787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc000cd790 0x7fbc000d16b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009870 tx=0x7fbbe802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283eee78d63944e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:55.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.155+0000 7fbbfdd9b640 1 -- 172.21.15.16:0/3336326787 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8037070 con 0x7fbc000cd790 2024-04-24T23:49:55.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.155+0000 7fbbfdd9b640 1 -- 172.21.15.16:0/3336326787 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe8032440 con 0x7fbc000cd790 2024-04-24T23:49:55.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.155+0000 7fbbfdd9b640 1 -- 172.21.15.16:0/3336326787 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8032810 con 0x7fbc000cd790 2024-04-24T23:49:55.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.156+0000 7fbc04c9e640 1 -- 172.21.15.16:0/3336326787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc000cd790 msgr2=0x7fbc000d16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.156+0000 7fbc04c9e640 1 --2- 172.21.15.16:0/3336326787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc000cd790 0x7fbc000d16b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009870 tx=0x7fbbe802a130 comp rx=0 tx=0).stop 2024-04-24T23:49:55.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.156+0000 7fbc04c9e640 1 -- 172.21.15.16:0/3336326787 shutdown_connections 2024-04-24T23:49:55.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.156+0000 7fbc04c9e640 1 --2- 172.21.15.16:0/3336326787 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbc0015e5c0 0x7fbc000d2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.156+0000 7fbc04c9e640 1 --2- 172.21.15.16:0/3336326787 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc000cd790 0x7fbc000d16b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.156+0000 7fbc04c9e640 1 -- 172.21.15.16:0/3336326787 >> 172.21.15.16:0/3336326787 conn(0x7fbc00072e70 msgr2=0x7fbc00073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.156+0000 7fbc04c9e640 1 -- 172.21.15.16:0/3336326787 shutdown_connections 2024-04-24T23:49:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.156+0000 7fbc04c9e640 1 -- 172.21.15.16:0/3336326787 wait complete. 2024-04-24T23:49:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.157+0000 7fbc04c9e640 1 Processor -- start 2024-04-24T23:49:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.157+0000 7fbc04c9e640 1 -- start start 2024-04-24T23:49:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbc04c9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc000cd790 0x7fbc00154670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbc04c9e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc000d12d0 0x7fbc000b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbc04c9e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbc0015e5c0 0x7fbc000b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbc04c9e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbc000d27f0 con 0x7fbc000cd790 2024-04-24T23:49:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbc04c9e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbc000d2670 con 0x7fbc000d12d0 2024-04-24T23:49:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbc04c9e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fbc000d2970 con 0x7fbc0015e5c0 2024-04-24T23:49:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbbfe59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc000cd790 0x7fbc00154670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbbfe59c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc000cd790 0x7fbc00154670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40904/0 (socket says 172.21.15.16:40904) 2024-04-24T23:49:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbbfe59c640 1 -- 172.21.15.16:0/1609259489 learned_addr learned my addr 172.21.15.16:0/1609259489 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbbf7fff640 1 --2- 172.21.15.16:0/1609259489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc000d12d0 0x7fbc000b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbbfed9d640 1 --2- 172.21.15.16:0/1609259489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbc0015e5c0 0x7fbc000b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbbfe59c640 1 -- 172.21.15.16:0/1609259489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbc0015e5c0 msgr2=0x7fbc000b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbbfe59c640 1 --2- 172.21.15.16:0/1609259489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbc0015e5c0 0x7fbc000b28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbbfe59c640 1 -- 172.21.15.16:0/1609259489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc000d12d0 msgr2=0x7fbc000b2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbbfe59c640 1 --2- 172.21.15.16:0/1609259489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc000d12d0 0x7fbc000b2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbbfe59c640 1 -- 172.21.15.16:0/1609259489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc001f6f70 con 0x7fbc000cd790 2024-04-24T23:49:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.158+0000 7fbbfe59c640 1 --2- 172.21.15.16:0/1609259489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc000cd790 0x7fbc00154670 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000c3c0 tx=0x7fbbf000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.159+0000 7fbbf6ffd640 1 -- 172.21.15.16:0/1609259489 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf00052d0 con 0x7fbc000cd790 2024-04-24T23:49:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.159+0000 7fbbf6ffd640 1 -- 172.21.15.16:0/1609259489 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbf0005490 con 0x7fbc000cd790 2024-04-24T23:49:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.159+0000 7fbc04c9e640 1 -- 172.21.15.16:0/1609259489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc001f7250 con 0x7fbc000cd790 2024-04-24T23:49:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.159+0000 7fbbf6ffd640 1 -- 172.21.15.16:0/1609259489 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbf0007260 con 0x7fbc000cd790 2024-04-24T23:49:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.159+0000 7fbc04c9e640 1 -- 172.21.15.16:0/1609259489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc001f7810 con 0x7fbc000cd790 2024-04-24T23:49:55.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.160+0000 7fbc04c9e640 1 -- 172.21.15.16:0/1609259489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc001f7b40 con 0x7fbc000cd790 2024-04-24T23:49:55.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.160+0000 7fbbf6ffd640 1 -- 172.21.15.16:0/1609259489 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbbf0007400 con 0x7fbc000cd790 2024-04-24T23:49:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.161+0000 7fbbf6ffd640 1 --2- 172.21.15.16:0/1609259489 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbbe006be30 0x7fbbe006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.161+0000 7fbbf6ffd640 1 -- 172.21.15.16:0/1609259489 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fbbf008a600 con 0x7fbc000cd790 2024-04-24T23:49:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.161+0000 7fbbf7fff640 1 --2- 172.21.15.16:0/1609259489 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbbe006be30 0x7fbbe006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.161+0000 7fbbf7fff640 1 --2- 172.21.15.16:0/1609259489 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbbe006be30 0x7fbbe006e2f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8034000 tx=0x7fbbe802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:55.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.165+0000 7fbbf6ffd640 1 -- 172.21.15.16:0/1609259489 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbbf00585f0 con 0x7fbc000cd790 2024-04-24T23:49:55.349 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.349+0000 7fbc04c9e640 1 -- 172.21.15.16:0/1609259489 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbc00068f90 con 0x7fbc000cd790 2024-04-24T23:49:55.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.350+0000 7fbbf6ffd640 1 -- 172.21.15.16:0/1609259489 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7fbbf005c8f0 con 0x7fbc000cd790 2024-04-24T23:49:55.350 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:55.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.352+0000 7fbc04c9e640 1 -- 172.21.15.16:0/1609259489 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbbe006be30 msgr2=0x7fbbe006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.352+0000 7fbc04c9e640 1 --2- 172.21.15.16:0/1609259489 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbbe006be30 0x7fbbe006e2f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8034000 tx=0x7fbbe802e000 comp rx=0 tx=0).stop 2024-04-24T23:49:55.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.352+0000 7fbc04c9e640 1 -- 172.21.15.16:0/1609259489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc000cd790 msgr2=0x7fbc00154670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.352+0000 7fbc04c9e640 1 --2- 172.21.15.16:0/1609259489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc000cd790 0x7fbc00154670 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbbf000c3c0 tx=0x7fbbf000c880 comp rx=0 tx=0).stop 2024-04-24T23:49:55.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.352+0000 7fbc04c9e640 1 -- 172.21.15.16:0/1609259489 shutdown_connections 2024-04-24T23:49:55.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.352+0000 7fbc04c9e640 1 --2- 172.21.15.16:0/1609259489 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbbe006be30 0x7fbbe006e2f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.352+0000 7fbc04c9e640 1 --2- 172.21.15.16:0/1609259489 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbc0015e5c0 0x7fbc000b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.352+0000 7fbc04c9e640 1 --2- 172.21.15.16:0/1609259489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc000d12d0 0x7fbc000b2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.352+0000 7fbc04c9e640 1 --2- 172.21.15.16:0/1609259489 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc000cd790 0x7fbc00154670 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.352+0000 7fbc04c9e640 1 -- 172.21.15.16:0/1609259489 >> 172.21.15.16:0/1609259489 conn(0x7fbc00072e70 msgr2=0x7fbc00073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:55.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.352+0000 7fbc04c9e640 1 -- 172.21.15.16:0/1609259489 shutdown_connections 2024-04-24T23:49:55.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.352+0000 7fbc04c9e640 1 -- 172.21.15.16:0/1609259489 wait complete. 2024-04-24T23:49:55.363 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.466+0000 7f7534b89640 1 Processor -- start 2024-04-24T23:49:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.466+0000 7f7534b89640 1 -- start start 2024-04-24T23:49:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.467+0000 7f7534b89640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f75300b6f90 0x7f75300b7410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.467+0000 7f7534b89640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75300d2940 0x7f75300d2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.467+0000 7f7534b89640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f753015e450 con 0x7f753015e070 2024-04-24T23:49:55.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.467+0000 7f7534b89640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f75300cf3a0 con 0x7f75300d2940 2024-04-24T23:49:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.467+0000 7f7534b89640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f753006ca40 con 0x7f75300b6f90 2024-04-24T23:49:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.467+0000 7f752e59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f753015e070 0x7f753006c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42374/0 (socket says 172.21.15.16:42374) 2024-04-24T23:49:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.467+0000 7f752e59c640 1 -- 172.21.15.16:0/270814806 learned_addr learned my addr 172.21.15.16:0/270814806 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.467+0000 7f752ed9d640 1 --2- 172.21.15.16:0/270814806 >> v2:172.21.15.16:3300/0 conn(0x7f75300d2940 0x7f75300d2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.467+0000 7f752dd9b640 1 --2- 172.21.15.16:0/270814806 >> v2:172.21.15.156:3300/0 conn(0x7f75300b6f90 0x7f75300b7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.468+0000 7f752d59a640 1 -- 172.21.15.16:0/270814806 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1026879274 0 0) 0x7f753015e450 con 0x7f753015e070 2024-04-24T23:49:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.468+0000 7f752d59a640 1 -- 172.21.15.16:0/270814806 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f751c003520 con 0x7f753015e070 2024-04-24T23:49:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.468+0000 7f752ed9d640 1 -- 172.21.15.16:0/270814806 >> v2:172.21.15.156:3300/0 conn(0x7f75300b6f90 msgr2=0x7f75300b7410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.468+0000 7f752ed9d640 1 --2- 172.21.15.16:0/270814806 >> v2:172.21.15.156:3300/0 conn(0x7f75300b6f90 0x7f75300b7410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.468+0000 7f752ed9d640 1 -- 172.21.15.16:0/270814806 >> v1:172.21.15.16:6789/0 conn(0x7f753015e070 legacy=0x7f753006c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.468+0000 7f752ed9d640 1 -- 172.21.15.16:0/270814806 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75300ce790 con 0x7f75300d2940 2024-04-24T23:49:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.468+0000 7f752ed9d640 1 --2- 172.21.15.16:0/270814806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75300d2940 0x7f75300d2dd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7530162c00 tx=0x7f752802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1d9f28e30222eca server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:55.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.469+0000 7f752d59a640 1 -- 172.21.15.16:0/270814806 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7528037070 con 0x7f75300d2940 2024-04-24T23:49:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.469+0000 7f752d59a640 1 -- 172.21.15.16:0/270814806 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7528032440 con 0x7f75300d2940 2024-04-24T23:49:55.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.469+0000 7f752d59a640 1 -- 172.21.15.16:0/270814806 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f752802ae60 con 0x7f75300d2940 2024-04-24T23:49:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.469+0000 7f7534b89640 1 -- 172.21.15.16:0/270814806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75300d2940 msgr2=0x7f75300d2dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.469+0000 7f7534b89640 1 --2- 172.21.15.16:0/270814806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75300d2940 0x7f75300d2dd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7530162c00 tx=0x7f752802a110 comp rx=0 tx=0).stop 2024-04-24T23:49:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.469+0000 7f7534b89640 1 -- 172.21.15.16:0/270814806 shutdown_connections 2024-04-24T23:49:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.469+0000 7f7534b89640 1 --2- 172.21.15.16:0/270814806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75300d2940 0x7f75300d2dd0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.469+0000 7f7534b89640 1 --2- 172.21.15.16:0/270814806 >> v2:172.21.15.156:3300/0 conn(0x7f75300b6f90 0x7f75300b7410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.469+0000 7f7534b89640 1 -- 172.21.15.16:0/270814806 >> 172.21.15.16:0/270814806 conn(0x7f75300b5850 msgr2=0x7f75300b5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.469+0000 7f7534b89640 1 -- 172.21.15.16:0/270814806 shutdown_connections 2024-04-24T23:49:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.469+0000 7f7534b89640 1 -- 172.21.15.16:0/270814806 wait complete. 2024-04-24T23:49:55.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.470+0000 7f7534b89640 1 Processor -- start 2024-04-24T23:49:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.471+0000 7f7534b89640 1 -- start start 2024-04-24T23:49:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.471+0000 7f7534b89640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75300b6f90 0x7f753015fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.471+0000 7f7534b89640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75300d2940 0x7f75300cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.471+0000 7f7534b89640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f753015e070 0x7f753014fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.471+0000 7f7534b89640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f75300cf030 con 0x7f75300d2940 2024-04-24T23:49:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.471+0000 7f7534b89640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f75300ced50 con 0x7f75300b6f90 2024-04-24T23:49:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.471+0000 7f7534b89640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f75300ce3b0 con 0x7f753015e070 2024-04-24T23:49:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.471+0000 7f752dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75300d2940 0x7f75300cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.471+0000 7f752dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75300d2940 0x7f75300cb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40930/0 (socket says 172.21.15.16:40930) 2024-04-24T23:49:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.471+0000 7f752dd9b640 1 -- 172.21.15.16:0/401068768 learned_addr learned my addr 172.21.15.16:0/401068768 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.471+0000 7f752e59c640 1 --2- 172.21.15.16:0/401068768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75300b6f90 0x7f753015fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.471+0000 7f752ed9d640 1 --2- 172.21.15.16:0/401068768 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f753015e070 0x7f753014fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.472+0000 7f752dd9b640 1 -- 172.21.15.16:0/401068768 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f753015e070 msgr2=0x7f753014fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.472+0000 7f752dd9b640 1 --2- 172.21.15.16:0/401068768 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f753015e070 0x7f753014fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.472+0000 7f752dd9b640 1 -- 172.21.15.16:0/401068768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75300b6f90 msgr2=0x7f753015fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.472+0000 7f752dd9b640 1 --2- 172.21.15.16:0/401068768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75300b6f90 0x7f753015fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.472+0000 7f752dd9b640 1 -- 172.21.15.16:0/401068768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75300cae30 con 0x7f75300d2940 2024-04-24T23:49:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.472+0000 7f752dd9b640 1 --2- 172.21.15.16:0/401068768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75300d2940 0x7f75300cb8f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f751800e870 tx=0x7f751800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.472+0000 7f7526ffd640 1 -- 172.21.15.16:0/401068768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7518017070 con 0x7f75300d2940 2024-04-24T23:49:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.472+0000 7f7526ffd640 1 -- 172.21.15.16:0/401068768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f751800c940 con 0x7f75300d2940 2024-04-24T23:49:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.472+0000 7f7534b89640 1 -- 172.21.15.16:0/401068768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75300cc900 con 0x7f75300d2940 2024-04-24T23:49:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.472+0000 7f7526ffd640 1 -- 172.21.15.16:0/401068768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f751800cd30 con 0x7f75300d2940 2024-04-24T23:49:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.472+0000 7f7534b89640 1 -- 172.21.15.16:0/401068768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75300cb0f0 con 0x7f75300d2940 2024-04-24T23:49:55.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.473+0000 7f7534b89640 1 -- 172.21.15.16:0/401068768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7530068f90 con 0x7f75300d2940 2024-04-24T23:49:55.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.474+0000 7f7526ffd640 1 -- 172.21.15.16:0/401068768 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f751801a070 con 0x7f75300d2940 2024-04-24T23:49:55.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.474+0000 7f7526ffd640 1 --2- 172.21.15.16:0/401068768 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f751006be60 0x7f751006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.474+0000 7f7526ffd640 1 -- 172.21.15.16:0/401068768 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f751808afd0 con 0x7f75300d2940 2024-04-24T23:49:55.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.474+0000 7f752e59c640 1 --2- 172.21.15.16:0/401068768 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f751006be60 0x7f751006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.475+0000 7f752e59c640 1 --2- 172.21.15.16:0/401068768 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f751006be60 0x7f751006e320 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7520005c00 tx=0x7f7520009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:55.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.478+0000 7f7526ffd640 1 -- 172.21.15.16:0/401068768 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7518058fc0 con 0x7f75300d2940 2024-04-24T23:49:55.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.662+0000 7f7534b89640 1 -- 172.21.15.16:0/401068768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f753015cab0 con 0x7f75300d2940 2024-04-24T23:49:55.662 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.663+0000 7f7526ffd640 1 -- 172.21.15.16:0/401068768 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f751805d2c0 con 0x7f75300d2940 2024-04-24T23:49:55.662 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:55.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.664+0000 7f7534b89640 1 -- 172.21.15.16:0/401068768 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f751006be60 msgr2=0x7f751006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.664+0000 7f7534b89640 1 --2- 172.21.15.16:0/401068768 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f751006be60 0x7f751006e320 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7520005c00 tx=0x7f7520009040 comp rx=0 tx=0).stop 2024-04-24T23:49:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.664+0000 7f7534b89640 1 -- 172.21.15.16:0/401068768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75300d2940 msgr2=0x7f75300cb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.664+0000 7f7534b89640 1 --2- 172.21.15.16:0/401068768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75300d2940 0x7f75300cb8f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f751800e870 tx=0x7f751800ed30 comp rx=0 tx=0).stop 2024-04-24T23:49:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.665+0000 7f7534b89640 1 -- 172.21.15.16:0/401068768 shutdown_connections 2024-04-24T23:49:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.665+0000 7f7534b89640 1 --2- 172.21.15.16:0/401068768 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f751006be60 0x7f751006e320 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.665+0000 7f7534b89640 1 --2- 172.21.15.16:0/401068768 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f753015e070 0x7f753014fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.665+0000 7f7534b89640 1 --2- 172.21.15.16:0/401068768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75300d2940 0x7f75300cb8f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.665+0000 7f7534b89640 1 --2- 172.21.15.16:0/401068768 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75300b6f90 0x7f753015fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.665+0000 7f7534b89640 1 -- 172.21.15.16:0/401068768 >> 172.21.15.16:0/401068768 conn(0x7f75300b5850 msgr2=0x7f75300b92e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.665+0000 7f7534b89640 1 -- 172.21.15.16:0/401068768 shutdown_connections 2024-04-24T23:49:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.665+0000 7f7534b89640 1 -- 172.21.15.16:0/401068768 wait complete. 2024-04-24T23:49:55.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:49:55.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.779+0000 7f2df20e1640 1 Processor -- start 2024-04-24T23:49:55.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.779+0000 7f2df20e1640 1 -- start start 2024-04-24T23:49:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.780+0000 7f2df20e1640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2dec0ce5f0 0x7f2dec0cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.780+0000 7f2df20e1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2dec0ccad0 0x7f2dec0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.780+0000 7f2df20e1640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2dec0d05a0 con 0x7f2dec15dfa0 2024-04-24T23:49:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.780+0000 7f2df20e1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2dec0b7360 con 0x7f2dec0ccad0 2024-04-24T23:49:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.780+0000 7f2df20e1640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2dec0cc640 con 0x7f2dec0ce5f0 2024-04-24T23:49:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.780+0000 7f2debfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2dec0ccad0 0x7f2dec0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.780+0000 7f2debfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2dec0ccad0 0x7f2dec0ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40946/0 (socket says 172.21.15.16:40946) 2024-04-24T23:49:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.780+0000 7f2debfff640 1 -- 172.21.15.16:0/1295563575 learned_addr learned my addr 172.21.15.16:0/1295563575 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.780+0000 7f2deaffd640 1 --2- 172.21.15.16:0/1295563575 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2dec0ce5f0 0x7f2dec0cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.781+0000 7f2debfff640 1 -- 172.21.15.16:0/1295563575 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2dec0ce5f0 msgr2=0x7f2dec0cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.781+0000 7f2debfff640 1 --2- 172.21.15.16:0/1295563575 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2dec0ce5f0 0x7f2dec0cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.781+0000 7f2debfff640 1 -- 172.21.15.16:0/1295563575 >> v1:172.21.15.156:6789/0 conn(0x7f2dec15dfa0 legacy=0x7f2dec0d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.781+0000 7f2debfff640 1 -- 172.21.15.16:0/1295563575 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dec0d0880 con 0x7f2dec0ccad0 2024-04-24T23:49:55.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.781+0000 7f2debfff640 1 --2- 172.21.15.16:0/1295563575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dec0ccad0 0x7f2dec0ccf60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2de4009e50 tx=0x7f2de402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83e34f21716f4b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:55.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.782+0000 7f2dea7fc640 1 -- 172.21.15.16:0/1295563575 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de4036070 con 0x7f2dec0ccad0 2024-04-24T23:49:55.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.782+0000 7f2dea7fc640 1 -- 172.21.15.16:0/1295563575 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2de4031600 con 0x7f2dec0ccad0 2024-04-24T23:49:55.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.782+0000 7f2dea7fc640 1 -- 172.21.15.16:0/1295563575 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de40319d0 con 0x7f2dec0ccad0 2024-04-24T23:49:55.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.782+0000 7f2df20e1640 1 -- 172.21.15.16:0/1295563575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dec0ccad0 msgr2=0x7f2dec0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.782+0000 7f2df20e1640 1 --2- 172.21.15.16:0/1295563575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dec0ccad0 0x7f2dec0ccf60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f2de4009e50 tx=0x7f2de402a710 comp rx=0 tx=0).stop 2024-04-24T23:49:55.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.782+0000 7f2df20e1640 1 -- 172.21.15.16:0/1295563575 shutdown_connections 2024-04-24T23:49:55.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.782+0000 7f2df20e1640 1 --2- 172.21.15.16:0/1295563575 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2dec0ce5f0 0x7f2dec0cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.782+0000 7f2df20e1640 1 --2- 172.21.15.16:0/1295563575 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dec0ccad0 0x7f2dec0ccf60 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.782+0000 7f2df20e1640 1 -- 172.21.15.16:0/1295563575 >> 172.21.15.16:0/1295563575 conn(0x7f2dec06b4d0 msgr2=0x7f2dec0b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:55.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.782+0000 7f2df20e1640 1 -- 172.21.15.16:0/1295563575 shutdown_connections 2024-04-24T23:49:55.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.782+0000 7f2df20e1640 1 -- 172.21.15.16:0/1295563575 wait complete. 2024-04-24T23:49:55.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.783+0000 7f2df20e1640 1 Processor -- start 2024-04-24T23:49:55.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.784+0000 7f2df20e1640 1 -- start start 2024-04-24T23:49:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.784+0000 7f2df20e1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2dec0ccad0 0x7f2dec0c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.784+0000 7f2df20e1640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2dec0ce5f0 0x7f2dec151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.784+0000 7f2df20e1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dec15dfa0 0x7f2dec0b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.784+0000 7f2df20e1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2dec15e380 con 0x7f2dec15dfa0 2024-04-24T23:49:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.784+0000 7f2df20e1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2dec0cc2d0 con 0x7f2dec0ccad0 2024-04-24T23:49:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.784+0000 7f2df20e1640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2dec0d01c0 con 0x7f2dec0ce5f0 2024-04-24T23:49:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.784+0000 7f2debfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dec15dfa0 0x7f2dec0b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.784+0000 7f2debfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dec15dfa0 0x7f2dec0b2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40950/0 (socket says 172.21.15.16:40950) 2024-04-24T23:49:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.784+0000 7f2debfff640 1 -- 172.21.15.16:0/2101081175 learned_addr learned my addr 172.21.15.16:0/2101081175 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.785+0000 7f2deb7fe640 1 --2- 172.21.15.16:0/2101081175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2dec0ccad0 0x7f2dec0c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.785+0000 7f2deaffd640 1 --2- 172.21.15.16:0/2101081175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2dec0ce5f0 0x7f2dec151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.785+0000 7f2debfff640 1 -- 172.21.15.16:0/2101081175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2dec0ce5f0 msgr2=0x7f2dec151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.785+0000 7f2debfff640 1 --2- 172.21.15.16:0/2101081175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2dec0ce5f0 0x7f2dec151050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.785+0000 7f2debfff640 1 -- 172.21.15.16:0/2101081175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2dec0ccad0 msgr2=0x7f2dec0c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.785+0000 7f2debfff640 1 --2- 172.21.15.16:0/2101081175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2dec0ccad0 0x7f2dec0c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.785+0000 7f2debfff640 1 -- 172.21.15.16:0/2101081175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2dec0c9500 con 0x7f2dec15dfa0 2024-04-24T23:49:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.785+0000 7f2debfff640 1 --2- 172.21.15.16:0/2101081175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dec15dfa0 0x7f2dec0b2140 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2de4034040 tx=0x7f2de4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.785+0000 7f2dd3fff640 1 -- 172.21.15.16:0/2101081175 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de4036050 con 0x7f2dec15dfa0 2024-04-24T23:49:55.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.785+0000 7f2dd3fff640 1 -- 172.21.15.16:0/2101081175 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2de4042070 con 0x7f2dec15dfa0 2024-04-24T23:49:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.785+0000 7f2df20e1640 1 -- 172.21.15.16:0/2101081175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2dec0c97e0 con 0x7f2dec15dfa0 2024-04-24T23:49:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.785+0000 7f2dd3fff640 1 -- 172.21.15.16:0/2101081175 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2de4009510 con 0x7f2dec15dfa0 2024-04-24T23:49:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.785+0000 7f2df20e1640 1 -- 172.21.15.16:0/2101081175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2dec0c9da0 con 0x7f2dec15dfa0 2024-04-24T23:49:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.786+0000 7f2df20e1640 1 -- 172.21.15.16:0/2101081175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2dec06bea0 con 0x7f2dec15dfa0 2024-04-24T23:49:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.787+0000 7f2dd3fff640 1 -- 172.21.15.16:0/2101081175 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2de4007bf0 con 0x7f2dec15dfa0 2024-04-24T23:49:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.787+0000 7f2dd3fff640 1 --2- 172.21.15.16:0/2101081175 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2dd806be30 0x7f2dd806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.787+0000 7f2dd3fff640 1 -- 172.21.15.16:0/2101081175 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f2de40abf90 con 0x7f2dec15dfa0 2024-04-24T23:49:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.787+0000 7f2deb7fe640 1 --2- 172.21.15.16:0/2101081175 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2dd806be30 0x7f2dd806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:55.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.788+0000 7f2deb7fe640 1 --2- 172.21.15.16:0/2101081175 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2dd806be30 0x7f2dd806e2f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc0016e0 tx=0x7f2ddc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:55.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.792+0000 7f2dd3fff640 1 -- 172.21.15.16:0/2101081175 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2de4079e60 con 0x7f2dec15dfa0 2024-04-24T23:49:55.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.989+0000 7f2df20e1640 1 -- 172.21.15.16:0/2101081175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2dec04f5e0 con 0x7f2dec15dfa0 2024-04-24T23:49:55.989 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.989+0000 7f2dd3fff640 1 -- 172.21.15.16:0/2101081175 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f2de407e160 con 0x7f2dec15dfa0 2024-04-24T23:49:55.989 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-04-24T23:49:55.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.991+0000 7f2df20e1640 1 -- 172.21.15.16:0/2101081175 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2dd806be30 msgr2=0x7f2dd806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.991+0000 7f2df20e1640 1 --2- 172.21.15.16:0/2101081175 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2dd806be30 0x7f2dd806e2f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2ddc0016e0 tx=0x7f2ddc009040 comp rx=0 tx=0).stop 2024-04-24T23:49:55.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.991+0000 7f2df20e1640 1 -- 172.21.15.16:0/2101081175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dec15dfa0 msgr2=0x7f2dec0b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:55.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.991+0000 7f2df20e1640 1 --2- 172.21.15.16:0/2101081175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dec15dfa0 0x7f2dec0b2140 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2de4034040 tx=0x7f2de4004a40 comp rx=0 tx=0).stop 2024-04-24T23:49:55.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.991+0000 7f2df20e1640 1 -- 172.21.15.16:0/2101081175 shutdown_connections 2024-04-24T23:49:55.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.991+0000 7f2df20e1640 1 --2- 172.21.15.16:0/2101081175 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2dd806be30 0x7f2dd806e2f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.991+0000 7f2df20e1640 1 --2- 172.21.15.16:0/2101081175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dec15dfa0 0x7f2dec0b2140 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.991+0000 7f2df20e1640 1 --2- 172.21.15.16:0/2101081175 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2dec0ce5f0 0x7f2dec151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.991+0000 7f2df20e1640 1 --2- 172.21.15.16:0/2101081175 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2dec0ccad0 0x7f2dec0c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:55.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.991+0000 7f2df20e1640 1 -- 172.21.15.16:0/2101081175 >> 172.21.15.16:0/2101081175 conn(0x7f2dec06b4d0 msgr2=0x7f2dec0d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:55.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.992+0000 7f2df20e1640 1 -- 172.21.15.16:0/2101081175 shutdown_connections 2024-04-24T23:49:55.992 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:55.992+0000 7f2df20e1640 1 -- 172.21.15.16:0/2101081175 wait complete. 2024-04-24T23:49:56.002 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2024-04-24T23:49:56.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:56.002 DEBUG:teuthology.orchestra.run.smithi156:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-04-24T23:49:56.028 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:49:56.029 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:49:56.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:49:56.029 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-24T23:49:56.095 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:49:56.095 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:49:56.111 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:49:56.111 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:49:56.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:56.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.309+0000 7f7f7d9f9640 1 Processor -- start 2024-04-24T23:49:56.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.309+0000 7f7f7d9f9640 1 -- start start 2024-04-24T23:49:56.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f7d9f9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f7815dfa0 0x7f7f780d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.311 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f7d9f9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f780cdae0 0x7f7f780cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f7d9f9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f780d05a0 con 0x7f7f780ce5f0 2024-04-24T23:49:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f7d9f9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f7f780b7360 con 0x7f7f780cdae0 2024-04-24T23:49:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f7d9f9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7f780cc640 con 0x7f7f7815dfa0 2024-04-24T23:49:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f777fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f780cdae0 0x7f7f780cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f6ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7f780ce5f0 0x7f7f780b5f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42384/0 (socket says 172.21.15.16:42384) 2024-04-24T23:49:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f6ffff640 1 -- 172.21.15.16:0/1866538277 learned_addr learned my addr 172.21.15.16:0/1866538277 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f76ffd640 1 --2- 172.21.15.16:0/1866538277 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f7815dfa0 0x7f7f780d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f777fe640 1 -- 172.21.15.16:0/1866538277 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f7815dfa0 msgr2=0x7f7f780d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f777fe640 1 --2- 172.21.15.16:0/1866538277 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f7815dfa0 0x7f7f780d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f777fe640 1 -- 172.21.15.16:0/1866538277 >> v1:172.21.15.16:6789/0 conn(0x7f7f780ce5f0 legacy=0x7f7f780b5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f777fe640 1 -- 172.21.15.16:0/1866538277 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f780d0880 con 0x7f7f780cdae0 2024-04-24T23:49:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f76ffd640 1 --2- 172.21.15.16:0/1866538277 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f7815dfa0 0x7f7f780d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.310+0000 7f7f767fc640 1 -- 172.21.15.16:0/1866538277 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1418263754 0 0) 0x7f7f780d05a0 con 0x7f7f780ce5f0 2024-04-24T23:49:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.311+0000 7f7f777fe640 1 --2- 172.21.15.16:0/1866538277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f780cdae0 0x7f7f780cea70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7f780cc7c0 tx=0x7f7f7002a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d6cf450628482aa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.311+0000 7f7f767fc640 1 -- 172.21.15.16:0/1866538277 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7002ae60 con 0x7f7f780cdae0 2024-04-24T23:49:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.311+0000 7f7f767fc640 1 -- 172.21.15.16:0/1866538277 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f70007740 con 0x7f7f780cdae0 2024-04-24T23:49:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.311+0000 7f7f767fc640 1 -- 172.21.15.16:0/1866538277 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f70007b10 con 0x7f7f780cdae0 2024-04-24T23:49:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.312+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/1866538277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f780cdae0 msgr2=0x7f7f780cea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.312+0000 7f7f7d9f9640 1 --2- 172.21.15.16:0/1866538277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f780cdae0 0x7f7f780cea70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f7f780cc7c0 tx=0x7f7f7002a640 comp rx=0 tx=0).stop 2024-04-24T23:49:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.312+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/1866538277 shutdown_connections 2024-04-24T23:49:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.312+0000 7f7f7d9f9640 1 --2- 172.21.15.16:0/1866538277 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f780cdae0 0x7f7f780cea70 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.312+0000 7f7f7d9f9640 1 --2- 172.21.15.16:0/1866538277 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f7815dfa0 0x7f7f780d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.312+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/1866538277 >> 172.21.15.16:0/1866538277 conn(0x7f7f7806b4d0 msgr2=0x7f7f780b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.312+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/1866538277 shutdown_connections 2024-04-24T23:49:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.312+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/1866538277 wait complete. 2024-04-24T23:49:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.313+0000 7f7f7d9f9640 1 Processor -- start 2024-04-24T23:49:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.313+0000 7f7f7d9f9640 1 -- start start 2024-04-24T23:49:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f7d9f9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f780cdae0 0x7f7f781559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f7d9f9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f780ce5f0 0x7f7f78156dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f7d9f9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f7815dfa0 0x7f7f78153020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f7d9f9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7f7815e380 con 0x7f7f780ce5f0 2024-04-24T23:49:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f7d9f9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7f780cc2d0 con 0x7f7f780cdae0 2024-04-24T23:49:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f7d9f9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7f780d1520 con 0x7f7f7815dfa0 2024-04-24T23:49:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f6ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f780ce5f0 0x7f7f78156dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f6ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f780ce5f0 0x7f7f78156dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40968/0 (socket says 172.21.15.16:40968) 2024-04-24T23:49:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f6ffff640 1 -- 172.21.15.16:0/3539730047 learned_addr learned my addr 172.21.15.16:0/3539730047 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f777fe640 1 --2- 172.21.15.16:0/3539730047 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f7815dfa0 0x7f7f78153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f6ffff640 1 -- 172.21.15.16:0/3539730047 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f7815dfa0 msgr2=0x7f7f78153020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f76ffd640 1 --2- 172.21.15.16:0/3539730047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f780cdae0 0x7f7f781559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f6ffff640 1 --2- 172.21.15.16:0/3539730047 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f7815dfa0 0x7f7f78153020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f6ffff640 1 -- 172.21.15.16:0/3539730047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f780cdae0 msgr2=0x7f7f781559f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f6ffff640 1 --2- 172.21.15.16:0/3539730047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f780cdae0 0x7f7f781559f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f6ffff640 1 -- 172.21.15.16:0/3539730047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f78157310 con 0x7f7f780ce5f0 2024-04-24T23:49:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f6ffff640 1 --2- 172.21.15.16:0/3539730047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f780ce5f0 0x7f7f78156dd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7f6000c4d0 tx=0x7f7f6000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f74ff9640 1 -- 172.21.15.16:0/3539730047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f60018070 con 0x7f7f780ce5f0 2024-04-24T23:49:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f74ff9640 1 -- 172.21.15.16:0/3539730047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f60007500 con 0x7f7f780ce5f0 2024-04-24T23:49:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/3539730047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f78156040 con 0x7f7f780ce5f0 2024-04-24T23:49:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.314+0000 7f7f74ff9640 1 -- 172.21.15.16:0/3539730047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f60004a20 con 0x7f7f780ce5f0 2024-04-24T23:49:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.315+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/3539730047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f78157890 con 0x7f7f780ce5f0 2024-04-24T23:49:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.316+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/3539730047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f7806bea0 con 0x7f7f780ce5f0 2024-04-24T23:49:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.316+0000 7f7f74ff9640 1 -- 172.21.15.16:0/3539730047 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7f6001b070 con 0x7f7f780ce5f0 2024-04-24T23:49:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.316+0000 7f7f74ff9640 1 --2- 172.21.15.16:0/3539730047 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7f5806be30 0x7f7f5806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.317+0000 7f7f74ff9640 1 -- 172.21.15.16:0/3539730047 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f7f6008a5a0 con 0x7f7f780ce5f0 2024-04-24T23:49:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.317+0000 7f7f76ffd640 1 --2- 172.21.15.16:0/3539730047 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7f5806be30 0x7f7f5806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.317+0000 7f7f76ffd640 1 --2- 172.21.15.16:0/3539730047 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7f5806be30 0x7f7f5806e2f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7f680096d0 tx=0x7f7f68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.321+0000 7f7f74ff9640 1 -- 172.21.15.16:0/3539730047 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7f78157310 con 0x7f7f780ce5f0 2024-04-24T23:49:56.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.508+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/3539730047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7f7804f5e0 con 0x7f7f780ce5f0 2024-04-24T23:49:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.509+0000 7f7f74ff9640 1 -- 172.21.15.16:0/3539730047 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7f7f60058b40 con 0x7f7f780ce5f0 2024-04-24T23:49:56.509 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:56.509 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.511+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/3539730047 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7f5806be30 msgr2=0x7f7f5806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.511+0000 7f7f7d9f9640 1 --2- 172.21.15.16:0/3539730047 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7f5806be30 0x7f7f5806e2f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7f680096d0 tx=0x7f7f68008040 comp rx=0 tx=0).stop 2024-04-24T23:49:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.511+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/3539730047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f780ce5f0 msgr2=0x7f7f78156dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.511+0000 7f7f7d9f9640 1 --2- 172.21.15.16:0/3539730047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f780ce5f0 0x7f7f78156dd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7f6000c4d0 tx=0x7f7f6000c990 comp rx=0 tx=0).stop 2024-04-24T23:49:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.511+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/3539730047 shutdown_connections 2024-04-24T23:49:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.511+0000 7f7f7d9f9640 1 --2- 172.21.15.16:0/3539730047 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7f5806be30 0x7f7f5806e2f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.511+0000 7f7f7d9f9640 1 --2- 172.21.15.16:0/3539730047 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7f7815dfa0 0x7f7f78153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.511+0000 7f7f7d9f9640 1 --2- 172.21.15.16:0/3539730047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f780ce5f0 0x7f7f78156dd0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.511+0000 7f7f7d9f9640 1 --2- 172.21.15.16:0/3539730047 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7f780cdae0 0x7f7f781559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.511+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/3539730047 >> 172.21.15.16:0/3539730047 conn(0x7f7f7806b4d0 msgr2=0x7f7f780b5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.511+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/3539730047 shutdown_connections 2024-04-24T23:49:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.511+0000 7f7f7d9f9640 1 -- 172.21.15.16:0/3539730047 wait complete. 2024-04-24T23:49:56.522 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:56.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:56.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.626+0000 7fbb08de9640 1 Processor -- start 2024-04-24T23:49:56.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.626+0000 7fbb08de9640 1 -- start start 2024-04-24T23:49:56.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.627+0000 7fbb08de9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbb0406bc40 0x7fbb0406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.627+0000 7fbb08de9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbb040b7050 0x7fbb040b7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.627+0000 7fbb08de9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb040d24d0 con 0x7fbb0415dfd0 2024-04-24T23:49:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.627+0000 7fbb08de9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fbb0406ba00 con 0x7fbb0406bc40 2024-04-24T23:49:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.627+0000 7fbb08de9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fbb040d27b0 con 0x7fbb040b7050 2024-04-24T23:49:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.627+0000 7fbb01d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbb0406bc40 0x7fbb0406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.627+0000 7fbb0259c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbb0415dfd0 0x7fbb0406b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42398/0 (socket says 172.21.15.16:42398) 2024-04-24T23:49:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.627+0000 7fbb0259c640 1 -- 172.21.15.16:0/996834655 learned_addr learned my addr 172.21.15.16:0/996834655 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.628+0000 7fbb02d9d640 1 --2- 172.21.15.16:0/996834655 >> v2:172.21.15.156:3300/0 conn(0x7fbb040b7050 0x7fbb040b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.628+0000 7fbb01d9b640 1 -- 172.21.15.16:0/996834655 >> v2:172.21.15.156:3300/0 conn(0x7fbb040b7050 msgr2=0x7fbb040b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.628+0000 7fbb01d9b640 1 --2- 172.21.15.16:0/996834655 >> v2:172.21.15.156:3300/0 conn(0x7fbb040b7050 0x7fbb040b7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.628+0000 7fbb01d9b640 1 -- 172.21.15.16:0/996834655 >> v1:172.21.15.16:6789/0 conn(0x7fbb0415dfd0 legacy=0x7fbb0406b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.628+0000 7fbb01d9b640 1 -- 172.21.15.16:0/996834655 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb040d2a90 con 0x7fbb0406bc40 2024-04-24T23:49:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.628+0000 7fbb0159a640 1 -- 172.21.15.16:0/996834655 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1281571755 0 0) 0x7fbb040d24d0 con 0x7fbb0415dfd0 2024-04-24T23:49:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.628+0000 7fbb01d9b640 1 --2- 172.21.15.16:0/996834655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb0406bc40 0x7fbb0406c0d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbb040b6ca0 tx=0x7fbaec02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90f6c76daf650e64 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.629+0000 7fbb0159a640 1 -- 172.21.15.16:0/996834655 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec037070 con 0x7fbb0406bc40 2024-04-24T23:49:56.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.629+0000 7fbb0159a640 1 -- 172.21.15.16:0/996834655 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaec032440 con 0x7fbb0406bc40 2024-04-24T23:49:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.629+0000 7fbb0159a640 1 -- 172.21.15.16:0/996834655 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaec02ae60 con 0x7fbb0406bc40 2024-04-24T23:49:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.629+0000 7fbb08de9640 1 -- 172.21.15.16:0/996834655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb0406bc40 msgr2=0x7fbb0406c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.629+0000 7fbb08de9640 1 --2- 172.21.15.16:0/996834655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb0406bc40 0x7fbb0406c0d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbb040b6ca0 tx=0x7fbaec02a110 comp rx=0 tx=0).stop 2024-04-24T23:49:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.629+0000 7fbb08de9640 1 -- 172.21.15.16:0/996834655 shutdown_connections 2024-04-24T23:49:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.629+0000 7fbb08de9640 1 --2- 172.21.15.16:0/996834655 >> v2:172.21.15.156:3300/0 conn(0x7fbb040b7050 0x7fbb040b7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.629+0000 7fbb08de9640 1 --2- 172.21.15.16:0/996834655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb0406bc40 0x7fbb0406c0d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.629+0000 7fbb08de9640 1 -- 172.21.15.16:0/996834655 >> 172.21.15.16:0/996834655 conn(0x7fbb040d3dc0 msgr2=0x7fbb040d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.629+0000 7fbb08de9640 1 -- 172.21.15.16:0/996834655 shutdown_connections 2024-04-24T23:49:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.629+0000 7fbb08de9640 1 -- 172.21.15.16:0/996834655 wait complete. 2024-04-24T23:49:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.631+0000 7fbb08de9640 1 Processor -- start 2024-04-24T23:49:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.631+0000 7fbb08de9640 1 -- start start 2024-04-24T23:49:56.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.631+0000 7fbb08de9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb0406bc40 0x7fbb040cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.631+0000 7fbb08de9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb040b7050 0x7fbb040c98b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.631+0000 7fbb08de9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb0415dfd0 0x7fbb040cae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.631+0000 7fbb08de9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbb040cead0 con 0x7fbb0406bc40 2024-04-24T23:49:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.631+0000 7fbb08de9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbb040d3050 con 0x7fbb0415dfd0 2024-04-24T23:49:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.631+0000 7fbb08de9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fbb040b79a0 con 0x7fbb040b7050 2024-04-24T23:49:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.631+0000 7fbb0259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb0406bc40 0x7fbb040cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.632+0000 7fbb0259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb0406bc40 0x7fbb040cbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40982/0 (socket says 172.21.15.16:40982) 2024-04-24T23:49:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.632+0000 7fbb0259c640 1 -- 172.21.15.16:0/713759060 learned_addr learned my addr 172.21.15.16:0/713759060 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.632+0000 7fbb02d9d640 1 --2- 172.21.15.16:0/713759060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb0415dfd0 0x7fbb040cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.632+0000 7fbb01d9b640 1 --2- 172.21.15.16:0/713759060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb040b7050 0x7fbb040c98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.632+0000 7fbb0259c640 1 -- 172.21.15.16:0/713759060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb040b7050 msgr2=0x7fbb040c98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.632+0000 7fbb0259c640 1 --2- 172.21.15.16:0/713759060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb040b7050 0x7fbb040c98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.632+0000 7fbb0259c640 1 -- 172.21.15.16:0/713759060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb0415dfd0 msgr2=0x7fbb040cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.632+0000 7fbb0259c640 1 --2- 172.21.15.16:0/713759060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb0415dfd0 0x7fbb040cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.632+0000 7fbb0259c640 1 -- 172.21.15.16:0/713759060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb040cc6e0 con 0x7fbb0406bc40 2024-04-24T23:49:56.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.632+0000 7fbb0259c640 1 --2- 172.21.15.16:0/713759060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb0406bc40 0x7fbb040cbe90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbafc00c430 tx=0x7fbafc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:56.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.632+0000 7fbafaffd640 1 -- 172.21.15.16:0/713759060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc0074e0 con 0x7fbb0406bc40 2024-04-24T23:49:56.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.633+0000 7fbafaffd640 1 -- 172.21.15.16:0/713759060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbafc0047e0 con 0x7fbb0406bc40 2024-04-24T23:49:56.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.633+0000 7fbb08de9640 1 -- 172.21.15.16:0/713759060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb04159040 con 0x7fbb0406bc40 2024-04-24T23:49:56.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.633+0000 7fbafaffd640 1 -- 172.21.15.16:0/713759060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc004bd0 con 0x7fbb0406bc40 2024-04-24T23:49:56.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.633+0000 7fbb08de9640 1 -- 172.21.15.16:0/713759060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb040cc3d0 con 0x7fbb0406bc40 2024-04-24T23:49:56.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.633+0000 7fbb08de9640 1 -- 172.21.15.16:0/713759060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb04068f90 con 0x7fbb0406bc40 2024-04-24T23:49:56.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.634+0000 7fbafaffd640 1 -- 172.21.15.16:0/713759060 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbafc011070 con 0x7fbb0406bc40 2024-04-24T23:49:56.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.634+0000 7fbafaffd640 1 --2- 172.21.15.16:0/713759060 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbae406be30 0x7fbae406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.634+0000 7fbafaffd640 1 -- 172.21.15.16:0/713759060 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fbafc08af00 con 0x7fbb0406bc40 2024-04-24T23:49:56.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.635+0000 7fbb01d9b640 1 --2- 172.21.15.16:0/713759060 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbae406be30 0x7fbae406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.635+0000 7fbb01d9b640 1 --2- 172.21.15.16:0/713759060 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbae406be30 0x7fbae406e2f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbaec034000 tx=0x7fbaec02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:56.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.639+0000 7fbafaffd640 1 -- 172.21.15.16:0/713759060 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbafc0594b0 con 0x7fbb0406bc40 2024-04-24T23:49:56.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.820+0000 7fbb08de9640 1 -- 172.21.15.16:0/713759060 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbb040cbc70 con 0x7fbb0406bc40 2024-04-24T23:49:56.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.821+0000 7fbafaffd640 1 -- 172.21.15.16:0/713759060 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fbafc058db0 con 0x7fbb0406bc40 2024-04-24T23:49:56.821 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.823+0000 7fbb08de9640 1 -- 172.21.15.16:0/713759060 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbae406be30 msgr2=0x7fbae406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.823+0000 7fbb08de9640 1 --2- 172.21.15.16:0/713759060 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbae406be30 0x7fbae406e2f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fbaec034000 tx=0x7fbaec02e000 comp rx=0 tx=0).stop 2024-04-24T23:49:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.823+0000 7fbb08de9640 1 -- 172.21.15.16:0/713759060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb0406bc40 msgr2=0x7fbb040cbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.823+0000 7fbb08de9640 1 --2- 172.21.15.16:0/713759060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb0406bc40 0x7fbb040cbe90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbafc00c430 tx=0x7fbafc00c8f0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.823+0000 7fbb08de9640 1 -- 172.21.15.16:0/713759060 shutdown_connections 2024-04-24T23:49:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.823+0000 7fbb08de9640 1 --2- 172.21.15.16:0/713759060 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbae406be30 0x7fbae406e2f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.823+0000 7fbb08de9640 1 --2- 172.21.15.16:0/713759060 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb0415dfd0 0x7fbb040cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.823+0000 7fbb08de9640 1 --2- 172.21.15.16:0/713759060 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbb040b7050 0x7fbb040c98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.823+0000 7fbb08de9640 1 --2- 172.21.15.16:0/713759060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb0406bc40 0x7fbb040cbe90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.823+0000 7fbb08de9640 1 -- 172.21.15.16:0/713759060 >> 172.21.15.16:0/713759060 conn(0x7fbb040d3dc0 msgr2=0x7fbb040b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:56.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.823+0000 7fbb08de9640 1 -- 172.21.15.16:0/713759060 shutdown_connections 2024-04-24T23:49:56.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.823+0000 7fbb08de9640 1 -- 172.21.15.16:0/713759060 wait complete. 2024-04-24T23:49:56.833 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:56.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.938+0000 7f77664ca640 1 Processor -- start 2024-04-24T23:49:56.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.938+0000 7f77664ca640 1 -- start start 2024-04-24T23:49:56.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.938+0000 7f77664ca640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f776015d0a0 0x7f77600ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.939 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.938+0000 7f77664ca640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f77600ceca0 0x7f776006ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.938+0000 7f77664ca640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f776015f9c0 con 0x7f77601455f0 2024-04-24T23:49:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.938+0000 7f77664ca640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f77601605d0 con 0x7f776015d0a0 2024-04-24T23:49:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.938+0000 7f77664ca640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f776015fca0 con 0x7f77600ceca0 2024-04-24T23:49:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.939+0000 7f775f7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f776015d0a0 0x7f77600ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.939+0000 7f775f7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f776015d0a0 0x7f77600ce760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56526/0 (socket says 172.21.15.16:56526) 2024-04-24T23:49:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.939+0000 7f775f7fe640 1 -- 172.21.15.16:0/3247183915 learned_addr learned my addr 172.21.15.16:0/3247183915 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.939+0000 7f7764a67640 1 --2- 172.21.15.16:0/3247183915 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f77600ceca0 0x7f776006ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.939+0000 7f775f7fe640 1 -- 172.21.15.16:0/3247183915 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f77600ceca0 msgr2=0x7f776006ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.939+0000 7f775f7fe640 1 --2- 172.21.15.16:0/3247183915 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f77600ceca0 0x7f776006ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.939+0000 7f775f7fe640 1 -- 172.21.15.16:0/3247183915 >> v1:172.21.15.156:6789/0 conn(0x7f77601455f0 legacy=0x7f776006b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.939+0000 7f775f7fe640 1 -- 172.21.15.16:0/3247183915 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f776015ff80 con 0x7f776015d0a0 2024-04-24T23:49:56.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.940+0000 7f775f7fe640 1 --2- 172.21.15.16:0/3247183915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f776015d0a0 0x7f77600ce760 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7754009870 tx=0x7f775402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9521303d187e0d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.940+0000 7f775effd640 1 -- 172.21.15.16:0/3247183915 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7754037070 con 0x7f776015d0a0 2024-04-24T23:49:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.940+0000 7f775effd640 1 -- 172.21.15.16:0/3247183915 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7754032440 con 0x7f776015d0a0 2024-04-24T23:49:56.941 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.940+0000 7f775effd640 1 -- 172.21.15.16:0/3247183915 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7754032810 con 0x7f776015d0a0 2024-04-24T23:49:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.940+0000 7f77664ca640 1 -- 172.21.15.16:0/3247183915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f776015d0a0 msgr2=0x7f77600ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.940+0000 7f77664ca640 1 --2- 172.21.15.16:0/3247183915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f776015d0a0 0x7f77600ce760 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7754009870 tx=0x7f775402a1c0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.940+0000 7f77664ca640 1 -- 172.21.15.16:0/3247183915 shutdown_connections 2024-04-24T23:49:56.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.940+0000 7f77664ca640 1 --2- 172.21.15.16:0/3247183915 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f77600ceca0 0x7f776006ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.940+0000 7f77664ca640 1 --2- 172.21.15.16:0/3247183915 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f776015d0a0 0x7f77600ce760 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.940+0000 7f77664ca640 1 -- 172.21.15.16:0/3247183915 >> 172.21.15.16:0/3247183915 conn(0x7f77600cfc50 msgr2=0x7f77600d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.940+0000 7f77664ca640 1 -- 172.21.15.16:0/3247183915 shutdown_connections 2024-04-24T23:49:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.940+0000 7f77664ca640 1 -- 172.21.15.16:0/3247183915 wait complete. 2024-04-24T23:49:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.942+0000 7f77664ca640 1 Processor -- start 2024-04-24T23:49:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.942+0000 7f77664ca640 1 -- start start 2024-04-24T23:49:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.942+0000 7f77664ca640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f77600ceca0 0x7f776013ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.942+0000 7f77664ca640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77601455f0 0x7f77600cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.942+0000 7f77664ca640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f776015d0a0 0x7f77601f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.942+0000 7f77664ca640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f77601459d0 con 0x7f77601455f0 2024-04-24T23:49:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.942+0000 7f77664ca640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f776006b7f0 con 0x7f776015d0a0 2024-04-24T23:49:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.942+0000 7f77664ca640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f776006d200 con 0x7f77600ceca0 2024-04-24T23:49:56.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f775f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77601455f0 0x7f77600cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f775f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77601455f0 0x7f77600cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56538/0 (socket says 172.21.15.16:56538) 2024-04-24T23:49:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f775f7fe640 1 -- 172.21.15.16:0/2642305186 learned_addr learned my addr 172.21.15.16:0/2642305186 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f775ffff640 1 --2- 172.21.15.16:0/2642305186 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f77600ceca0 0x7f776013ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f7764a67640 1 --2- 172.21.15.16:0/2642305186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f776015d0a0 0x7f77601f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f775f7fe640 1 -- 172.21.15.16:0/2642305186 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f77600ceca0 msgr2=0x7f776013ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f775f7fe640 1 --2- 172.21.15.16:0/2642305186 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f77600ceca0 0x7f776013ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f775f7fe640 1 -- 172.21.15.16:0/2642305186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f776015d0a0 msgr2=0x7f77601f2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f775f7fe640 1 --2- 172.21.15.16:0/2642305186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f776015d0a0 0x7f77601f2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f775f7fe640 1 -- 172.21.15.16:0/2642305186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f776013d370 con 0x7f77601455f0 2024-04-24T23:49:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f775f7fe640 1 --2- 172.21.15.16:0/2642305186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77601455f0 0x7f77600cbc40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f77540090b0 tx=0x7f77540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f775cff9640 1 -- 172.21.15.16:0/2642305186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7754041030 con 0x7f77601455f0 2024-04-24T23:49:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f775cff9640 1 -- 172.21.15.16:0/2642305186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7754030440 con 0x7f77601455f0 2024-04-24T23:49:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f77664ca640 1 -- 172.21.15.16:0/2642305186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f776013d650 con 0x7f77601455f0 2024-04-24T23:49:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f775cff9640 1 -- 172.21.15.16:0/2642305186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7754037040 con 0x7f77601455f0 2024-04-24T23:49:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.943+0000 7f77664ca640 1 -- 172.21.15.16:0/2642305186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f776013b070 con 0x7f77601455f0 2024-04-24T23:49:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.944+0000 7f77664ca640 1 -- 172.21.15.16:0/2642305186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7760068f90 con 0x7f77601455f0 2024-04-24T23:49:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.945+0000 7f775cff9640 1 -- 172.21.15.16:0/2642305186 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7754044070 con 0x7f77601455f0 2024-04-24T23:49:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.945+0000 7f775cff9640 1 --2- 172.21.15.16:0/2642305186 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f774c06be30 0x7f774c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.945+0000 7f775cff9640 1 -- 172.21.15.16:0/2642305186 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f77540ab4a0 con 0x7f77601455f0 2024-04-24T23:49:56.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.945+0000 7f775ffff640 1 --2- 172.21.15.16:0/2642305186 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f774c06be30 0x7f774c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:56.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.946+0000 7f775ffff640 1 --2- 172.21.15.16:0/2642305186 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f774c06be30 0x7f774c06e2f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f776013eeb0 tx=0x7f7750009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:56.949+0000 7f775cff9640 1 -- 172.21.15.16:0/2642305186 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f775404b070 con 0x7f77601455f0 2024-04-24T23:49:57.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.131+0000 7f77664ca640 1 -- 172.21.15.16:0/2642305186 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f77600b3150 con 0x7f77601455f0 2024-04-24T23:49:57.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.132+0000 7f775cff9640 1 -- 172.21.15.16:0/2642305186 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f7754079490 con 0x7f77601455f0 2024-04-24T23:49:57.132 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.134+0000 7f77664ca640 1 -- 172.21.15.16:0/2642305186 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f774c06be30 msgr2=0x7f774c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.134+0000 7f77664ca640 1 --2- 172.21.15.16:0/2642305186 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f774c06be30 0x7f774c06e2f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f776013eeb0 tx=0x7f7750009040 comp rx=0 tx=0).stop 2024-04-24T23:49:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.134+0000 7f77664ca640 1 -- 172.21.15.16:0/2642305186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77601455f0 msgr2=0x7f77600cbc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.134+0000 7f77664ca640 1 --2- 172.21.15.16:0/2642305186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77601455f0 0x7f77600cbc40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f77540090b0 tx=0x7f77540046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.134+0000 7f77664ca640 1 -- 172.21.15.16:0/2642305186 shutdown_connections 2024-04-24T23:49:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.134+0000 7f77664ca640 1 --2- 172.21.15.16:0/2642305186 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f774c06be30 0x7f774c06e2f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.134+0000 7f77664ca640 1 --2- 172.21.15.16:0/2642305186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f776015d0a0 0x7f77601f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.134+0000 7f77664ca640 1 --2- 172.21.15.16:0/2642305186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77601455f0 0x7f77600cbc40 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.134+0000 7f77664ca640 1 --2- 172.21.15.16:0/2642305186 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f77600ceca0 0x7f776013ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.134+0000 7f77664ca640 1 -- 172.21.15.16:0/2642305186 >> 172.21.15.16:0/2642305186 conn(0x7f77600cfc50 msgr2=0x7f776006de40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:57.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.134+0000 7f77664ca640 1 -- 172.21.15.16:0/2642305186 shutdown_connections 2024-04-24T23:49:57.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.134+0000 7f77664ca640 1 -- 172.21.15.16:0/2642305186 wait complete. 2024-04-24T23:49:57.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:57.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.249+0000 7f2b1fd63640 1 Processor -- start 2024-04-24T23:49:57.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.249+0000 7f2b1fd63640 1 -- start start 2024-04-24T23:49:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1fd63640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2b1815dff0 0x7f2b1806deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1fd63640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b1806d360 0x7f2b180af270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1fd63640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b180af7b0 0x7f2b180ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1fd63640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2b180afc10 con 0x7f2b180af7b0 2024-04-24T23:49:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1fd63640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f2b180732d0 con 0x7f2b1815dff0 2024-04-24T23:49:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1fd63640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2b1806d8d0 con 0x7f2b1806d360 2024-04-24T23:49:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1e300640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b180af7b0 0x7f2b180ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1e300640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b180af7b0 0x7f2b180ad8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56552/0 (socket says 172.21.15.16:56552) 2024-04-24T23:49:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1e300640 1 -- 172.21.15.16:0/3541399515 learned_addr learned my addr 172.21.15.16:0/3541399515 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1daff640 1 --2- 172.21.15.16:0/3541399515 >> v2:172.21.15.156:3300/0 conn(0x7f2b1815dff0 0x7f2b1806deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1d2fe640 1 --2- 172.21.15.16:0/3541399515 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b1806d360 0x7f2b180af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1e300640 1 -- 172.21.15.16:0/3541399515 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b1806d360 msgr2=0x7f2b180af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1e300640 1 --2- 172.21.15.16:0/3541399515 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b1806d360 0x7f2b180af270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1e300640 1 -- 172.21.15.16:0/3541399515 >> v2:172.21.15.156:3300/0 conn(0x7f2b1815dff0 msgr2=0x7f2b1806deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1e300640 1 --2- 172.21.15.16:0/3541399515 >> v2:172.21.15.156:3300/0 conn(0x7f2b1815dff0 0x7f2b1806deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.250+0000 7f2b1e300640 1 -- 172.21.15.16:0/3541399515 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1806dbb0 con 0x7f2b180af7b0 2024-04-24T23:49:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.251+0000 7f2b1e300640 1 --2- 172.21.15.16:0/3541399515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b180af7b0 0x7f2b180ad8e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2b18067ea0 tx=0x7f2b1002a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0360c4297e95b41 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.251+0000 7f2b1cafd640 1 -- 172.21.15.16:0/3541399515 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b10036070 con 0x7f2b180af7b0 2024-04-24T23:49:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.251+0000 7f2b1cafd640 1 -- 172.21.15.16:0/3541399515 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b10031600 con 0x7f2b180af7b0 2024-04-24T23:49:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.251+0000 7f2b1cafd640 1 -- 172.21.15.16:0/3541399515 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b100319d0 con 0x7f2b180af7b0 2024-04-24T23:49:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.252+0000 7f2b1fd63640 1 -- 172.21.15.16:0/3541399515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b180af7b0 msgr2=0x7f2b180ad8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.252+0000 7f2b1fd63640 1 --2- 172.21.15.16:0/3541399515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b180af7b0 0x7f2b180ad8e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f2b18067ea0 tx=0x7f2b1002a6c0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.252+0000 7f2b1fd63640 1 -- 172.21.15.16:0/3541399515 shutdown_connections 2024-04-24T23:49:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.252+0000 7f2b1fd63640 1 --2- 172.21.15.16:0/3541399515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b180af7b0 0x7f2b180ad8e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.252+0000 7f2b1fd63640 1 --2- 172.21.15.16:0/3541399515 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b1806d360 0x7f2b180af270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.252+0000 7f2b1fd63640 1 --2- 172.21.15.16:0/3541399515 >> v2:172.21.15.156:3300/0 conn(0x7f2b1815dff0 0x7f2b1806deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.252+0000 7f2b1fd63640 1 -- 172.21.15.16:0/3541399515 >> 172.21.15.16:0/3541399515 conn(0x7f2b180affc0 msgr2=0x7f2b180b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.252+0000 7f2b1fd63640 1 -- 172.21.15.16:0/3541399515 shutdown_connections 2024-04-24T23:49:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.252+0000 7f2b1fd63640 1 -- 172.21.15.16:0/3541399515 wait complete. 2024-04-24T23:49:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.253+0000 7f2b1fd63640 1 Processor -- start 2024-04-24T23:49:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.253+0000 7f2b1fd63640 1 -- start start 2024-04-24T23:49:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1fd63640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1806d360 0x7f2b18146570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1fd63640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b180af7b0 0x7f2b180b1200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1fd63640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b1815dff0 0x7f2b180b1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1fd63640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b180718f0 con 0x7f2b1806d360 2024-04-24T23:49:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1fd63640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2b18071610 con 0x7f2b180af7b0 2024-04-24T23:49:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1fd63640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2b180ccca0 con 0x7f2b1815dff0 2024-04-24T23:49:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1daff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1806d360 0x7f2b18146570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1daff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1806d360 0x7f2b18146570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56562/0 (socket says 172.21.15.16:56562) 2024-04-24T23:49:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1daff640 1 -- 172.21.15.16:0/566231634 learned_addr learned my addr 172.21.15.16:0/566231634 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1d2fe640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b180af7b0 0x7f2b180b1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1e300640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b1815dff0 0x7f2b180b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1daff640 1 -- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b1815dff0 msgr2=0x7f2b180b1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1daff640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b1815dff0 0x7f2b180b1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1daff640 1 -- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b180af7b0 msgr2=0x7f2b180b1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1daff640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b180af7b0 0x7f2b180b1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1daff640 1 -- 172.21.15.16:0/566231634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b181604a0 con 0x7f2b1806d360 2024-04-24T23:49:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1d2fe640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b180af7b0 0x7f2b180b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:57.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.254+0000 7f2b1daff640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1806d360 0x7f2b18146570 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400d5f0 tx=0x7f2b1400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.255+0000 7f2b0e7fc640 1 -- 172.21.15.16:0/566231634 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b140125f0 con 0x7f2b1806d360 2024-04-24T23:49:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.255+0000 7f2b0e7fc640 1 -- 172.21.15.16:0/566231634 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b14019070 con 0x7f2b1806d360 2024-04-24T23:49:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.255+0000 7f2b1fd63640 1 -- 172.21.15.16:0/566231634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b181601c0 con 0x7f2b1806d360 2024-04-24T23:49:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.255+0000 7f2b0e7fc640 1 -- 172.21.15.16:0/566231634 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b140129d0 con 0x7f2b1806d360 2024-04-24T23:49:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.255+0000 7f2b1fd63640 1 -- 172.21.15.16:0/566231634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b181470b0 con 0x7f2b1806d360 2024-04-24T23:49:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.255+0000 7f2b1fd63640 1 -- 172.21.15.16:0/566231634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b180af270 con 0x7f2b1806d360 2024-04-24T23:49:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.256+0000 7f2b0e7fc640 1 -- 172.21.15.16:0/566231634 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2b14012b70 con 0x7f2b1806d360 2024-04-24T23:49:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.256+0000 7f2b0e7fc640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2afc06be30 0x7f2afc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.257+0000 7f2b0e7fc640 1 -- 172.21.15.16:0/566231634 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f2b1408b680 con 0x7f2b1806d360 2024-04-24T23:49:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.257+0000 7f2b1d2fe640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2afc06be30 0x7f2afc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.257+0000 7f2b1d2fe640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2afc06be30 0x7f2afc06e2f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2b181614a0 tx=0x7f2b08008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:57.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.261+0000 7f2b0e7fc640 1 -- 172.21.15.16:0/566231634 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2b180af270 con 0x7f2b1806d360 2024-04-24T23:49:57.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.443+0000 7f2b1fd63640 1 -- 172.21.15.16:0/566231634 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2b18068f90 con 0x7f2b1806d360 2024-04-24T23:49:57.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.443+0000 7f2b0e7fc640 1 -- 172.21.15.16:0/566231634 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f2b14059ba0 con 0x7f2b1806d360 2024-04-24T23:49:57.452 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.445+0000 7f2b1fd63640 1 -- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2afc06be30 msgr2=0x7f2afc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.445+0000 7f2b1fd63640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2afc06be30 0x7f2afc06e2f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2b181614a0 tx=0x7f2b08008040 comp rx=0 tx=0).stop 2024-04-24T23:49:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.445+0000 7f2b1fd63640 1 -- 172.21.15.16:0/566231634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1806d360 msgr2=0x7f2b18146570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.445+0000 7f2b1fd63640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1806d360 0x7f2b18146570 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2b1400d5f0 tx=0x7f2b1400dab0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.445+0000 7f2b1fd63640 1 -- 172.21.15.16:0/566231634 shutdown_connections 2024-04-24T23:49:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.445+0000 7f2b1fd63640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2afc06be30 0x7f2afc06e2f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.445+0000 7f2b1fd63640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b1815dff0 0x7f2b180b1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.445+0000 7f2b1fd63640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b180af7b0 0x7f2b180b1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.445+0000 7f2b1fd63640 1 --2- 172.21.15.16:0/566231634 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1806d360 0x7f2b18146570 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.445+0000 7f2b1fd63640 1 -- 172.21.15.16:0/566231634 >> 172.21.15.16:0/566231634 conn(0x7f2b180affc0 msgr2=0x7f2b180cd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.445+0000 7f2b1fd63640 1 -- 172.21.15.16:0/566231634 shutdown_connections 2024-04-24T23:49:57.454 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.445+0000 7f2b1fd63640 1 -- 172.21.15.16:0/566231634 wait complete. 2024-04-24T23:49:57.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:57.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.560+0000 7f80c27dc640 1 Processor -- start 2024-04-24T23:49:57.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.560+0000 7f80c27dc640 1 -- start start 2024-04-24T23:49:57.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.561+0000 7f80c27dc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80bc0d12d0 0x7f80bc0d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.561+0000 7f80c27dc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f80bc0cd790 0x7f80bc15e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.561+0000 7f80c27dc640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80bc15ec30 con 0x7f80bc0c8580 2024-04-24T23:49:57.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.561+0000 7f80c27dc640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f80bc15f4d0 con 0x7f80bc0d12d0 2024-04-24T23:49:57.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.561+0000 7f80c27dc640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f80bc15ef10 con 0x7f80bc0cd790 2024-04-24T23:49:57.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.561+0000 7f80bbfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80bc0d12d0 0x7f80bc0d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.561+0000 7f80bbfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80bc0d12d0 0x7f80bc0d16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56578/0 (socket says 172.21.15.16:56578) 2024-04-24T23:49:57.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.561+0000 7f80bbfff640 1 -- 172.21.15.16:0/2265287524 learned_addr learned my addr 172.21.15.16:0/2265287524 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:57.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.561+0000 7f80bb7fe640 1 --2- 172.21.15.16:0/2265287524 >> v2:172.21.15.156:3300/0 conn(0x7f80bc0cd790 0x7f80bc15e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.562+0000 7f80bbfff640 1 -- 172.21.15.16:0/2265287524 >> v2:172.21.15.156:3300/0 conn(0x7f80bc0cd790 msgr2=0x7f80bc15e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:57.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.562+0000 7f80bbfff640 1 --2- 172.21.15.16:0/2265287524 >> v2:172.21.15.156:3300/0 conn(0x7f80bc0cd790 0x7f80bc15e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.562+0000 7f80bbfff640 1 -- 172.21.15.16:0/2265287524 >> v1:172.21.15.156:6789/0 conn(0x7f80bc0c8580 legacy=0x7f80bc0d2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:57.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.562+0000 7f80bbfff640 1 -- 172.21.15.16:0/2265287524 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80bc15f1f0 con 0x7f80bc0d12d0 2024-04-24T23:49:57.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.562+0000 7f80bbfff640 1 --2- 172.21.15.16:0/2265287524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80bc0d12d0 0x7f80bc0d16b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0d1bf0 tx=0x7f80b002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dff7bddc83dbec7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:57.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.562+0000 7f80baffd640 1 -- 172.21.15.16:0/2265287524 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b0037070 con 0x7f80bc0d12d0 2024-04-24T23:49:57.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.562+0000 7f80baffd640 1 -- 172.21.15.16:0/2265287524 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80b002ae20 con 0x7f80bc0d12d0 2024-04-24T23:49:57.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.562+0000 7f80baffd640 1 -- 172.21.15.16:0/2265287524 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b0032660 con 0x7f80bc0d12d0 2024-04-24T23:49:57.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.563+0000 7f80c27dc640 1 -- 172.21.15.16:0/2265287524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80bc0d12d0 msgr2=0x7f80bc0d16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.563+0000 7f80c27dc640 1 --2- 172.21.15.16:0/2265287524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80bc0d12d0 0x7f80bc0d16b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f80bc0d1bf0 tx=0x7f80b002a110 comp rx=0 tx=0).stop 2024-04-24T23:49:57.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.563+0000 7f80c27dc640 1 -- 172.21.15.16:0/2265287524 shutdown_connections 2024-04-24T23:49:57.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.563+0000 7f80c27dc640 1 --2- 172.21.15.16:0/2265287524 >> v2:172.21.15.156:3300/0 conn(0x7f80bc0cd790 0x7f80bc15e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.563+0000 7f80c27dc640 1 --2- 172.21.15.16:0/2265287524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80bc0d12d0 0x7f80bc0d16b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.563+0000 7f80c27dc640 1 -- 172.21.15.16:0/2265287524 >> 172.21.15.16:0/2265287524 conn(0x7f80bc072e70 msgr2=0x7f80bc073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:57.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.563+0000 7f80c27dc640 1 -- 172.21.15.16:0/2265287524 shutdown_connections 2024-04-24T23:49:57.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.563+0000 7f80c27dc640 1 -- 172.21.15.16:0/2265287524 wait complete. 2024-04-24T23:49:57.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.564+0000 7f80c27dc640 1 Processor -- start 2024-04-24T23:49:57.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.564+0000 7f80c27dc640 1 -- start start 2024-04-24T23:49:57.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.565+0000 7f80c27dc640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc0c8580 0x7f80bc157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.565+0000 7f80c27dc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80bc0cd790 0x7f80bc155550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.565+0000 7f80c27dc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f80bc0d12d0 0x7f80bc155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.565+0000 7f80c27dc640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f80bc0c8b50 con 0x7f80bc0cd790 2024-04-24T23:49:57.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.565+0000 7f80c27dc640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f80bc0c89d0 con 0x7f80bc0c8580 2024-04-24T23:49:57.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.565+0000 7f80c27dc640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f80bc0d2840 con 0x7f80bc0d12d0 2024-04-24T23:49:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.565+0000 7f80bb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80bc0cd790 0x7f80bc155550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.565+0000 7f80bb7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80bc0cd790 0x7f80bc155550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56586/0 (socket says 172.21.15.16:56586) 2024-04-24T23:49:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.565+0000 7f80bb7fe640 1 -- 172.21.15.16:0/1890549782 learned_addr learned my addr 172.21.15.16:0/1890549782 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.565+0000 7f80bbfff640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc0c8580 0x7f80bc157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.566+0000 7f80c0d79640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f80bc0d12d0 0x7f80bc155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.566+0000 7f80bb7fe640 1 -- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f80bc0d12d0 msgr2=0x7f80bc155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.566+0000 7f80bb7fe640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f80bc0d12d0 0x7f80bc155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.566+0000 7f80bb7fe640 1 -- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc0c8580 msgr2=0x7f80bc157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.566+0000 7f80bb7fe640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc0c8580 0x7f80bc157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.566+0000 7f80bb7fe640 1 -- 172.21.15.16:0/1890549782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80bc1560a0 con 0x7f80bc0cd790 2024-04-24T23:49:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.566+0000 7f80c0d79640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f80bc0d12d0 0x7f80bc155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.566+0000 7f80bb7fe640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80bc0cd790 0x7f80bc155550 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f80ac00d800 tx=0x7f80ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.566+0000 7f80b8ff9640 1 -- 172.21.15.16:0/1890549782 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac00b6c0 con 0x7f80bc0cd790 2024-04-24T23:49:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.566+0000 7f80b8ff9640 1 -- 172.21.15.16:0/1890549782 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80ac019070 con 0x7f80bc0cd790 2024-04-24T23:49:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.566+0000 7f80c27dc640 1 -- 172.21.15.16:0/1890549782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80bc156380 con 0x7f80bc0cd790 2024-04-24T23:49:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.566+0000 7f80b8ff9640 1 -- 172.21.15.16:0/1890549782 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80ac00ba80 con 0x7f80bc0cd790 2024-04-24T23:49:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.566+0000 7f80c27dc640 1 -- 172.21.15.16:0/1890549782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80bc156c20 con 0x7f80bc0cd790 2024-04-24T23:49:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.567+0000 7f80c27dc640 1 -- 172.21.15.16:0/1890549782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80bc0adbc0 con 0x7f80bc0cd790 2024-04-24T23:49:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.567+0000 7f80b8ff9640 1 -- 172.21.15.16:0/1890549782 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f80ac00bc20 con 0x7f80bc0cd790 2024-04-24T23:49:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.568+0000 7f80b8ff9640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f80a806be30 0x7f80a806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.568+0000 7f80b8ff9640 1 -- 172.21.15.16:0/1890549782 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f80ac08b6a0 con 0x7f80bc0cd790 2024-04-24T23:49:57.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.568+0000 7f80bbfff640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f80a806be30 0x7f80a806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.568+0000 7f80bbfff640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f80a806be30 0x7f80a806e2f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f80b0034000 tx=0x7f80b002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:57.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.572+0000 7f80b8ff9640 1 -- 172.21.15.16:0/1890549782 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f80ac01c360 con 0x7f80bc0cd790 2024-04-24T23:49:57.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.755+0000 7f80c27dc640 1 -- 172.21.15.16:0/1890549782 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f80bc068f90 con 0x7f80bc0cd790 2024-04-24T23:49:57.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.755+0000 7f80b8ff9640 1 -- 172.21.15.16:0/1890549782 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f80ac059ce0 con 0x7f80bc0cd790 2024-04-24T23:49:57.755 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.757+0000 7f80c27dc640 1 -- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f80a806be30 msgr2=0x7f80a806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.757+0000 7f80c27dc640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f80a806be30 0x7f80a806e2f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f80b0034000 tx=0x7f80b002e000 comp rx=0 tx=0).stop 2024-04-24T23:49:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.757+0000 7f80c27dc640 1 -- 172.21.15.16:0/1890549782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80bc0cd790 msgr2=0x7f80bc155550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.757+0000 7f80c27dc640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80bc0cd790 0x7f80bc155550 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f80ac00d800 tx=0x7f80ac00dcc0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.757+0000 7f80c27dc640 1 -- 172.21.15.16:0/1890549782 shutdown_connections 2024-04-24T23:49:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.757+0000 7f80c27dc640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f80a806be30 0x7f80a806e2f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.757+0000 7f80c27dc640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f80bc0d12d0 0x7f80bc155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.757+0000 7f80c27dc640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80bc0cd790 0x7f80bc155550 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.757+0000 7f80c27dc640 1 --2- 172.21.15.16:0/1890549782 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc0c8580 0x7f80bc157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.757+0000 7f80c27dc640 1 -- 172.21.15.16:0/1890549782 >> 172.21.15.16:0/1890549782 conn(0x7f80bc072e70 msgr2=0x7f80bc0d2130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.757+0000 7f80c27dc640 1 -- 172.21.15.16:0/1890549782 shutdown_connections 2024-04-24T23:49:57.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.757+0000 7f80c27dc640 1 -- 172.21.15.16:0/1890549782 wait complete. 2024-04-24T23:49:57.768 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:49:57.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.876+0000 7f1d101a9640 1 Processor -- start 2024-04-24T23:49:57.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.876+0000 7f1d101a9640 1 -- start start 2024-04-24T23:49:57.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.877+0000 7f1d101a9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1d0815dfa0 0x7f1d080d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.877+0000 7f1d101a9640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1d080cdae0 0x7f1d080cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.877+0000 7f1d101a9640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d080d05a0 con 0x7f1d080ce5f0 2024-04-24T23:49:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.877+0000 7f1d101a9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f1d080b7360 con 0x7f1d0815dfa0 2024-04-24T23:49:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.877+0000 7f1d101a9640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f1d080cc640 con 0x7f1d080cdae0 2024-04-24T23:49:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.877+0000 7f1d0df45640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1d0815dfa0 0x7f1d080d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.877+0000 7f1d0df45640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1d0815dfa0 0x7f1d080d0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56602/0 (socket says 172.21.15.16:56602) 2024-04-24T23:49:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.877+0000 7f1d0df45640 1 -- 172.21.15.16:0/3324622592 learned_addr learned my addr 172.21.15.16:0/3324622592 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.877+0000 7f1d0df45640 1 -- 172.21.15.16:0/3324622592 >> v2:172.21.15.156:3300/0 conn(0x7f1d080cdae0 msgr2=0x7f1d080cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.877+0000 7f1d0df45640 1 --2- 172.21.15.16:0/3324622592 >> v2:172.21.15.156:3300/0 conn(0x7f1d080cdae0 0x7f1d080cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.877+0000 7f1d0df45640 1 -- 172.21.15.16:0/3324622592 >> v1:172.21.15.156:6789/0 conn(0x7f1d080ce5f0 legacy=0x7f1d080b5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.877+0000 7f1d0df45640 1 -- 172.21.15.16:0/3324622592 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d080d0880 con 0x7f1d0815dfa0 2024-04-24T23:49:57.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.878+0000 7f1d0df45640 1 --2- 172.21.15.16:0/3324622592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d0815dfa0 0x7f1d080d0fe0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1d080cf3e0 tx=0x7f1d0002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeddbfc88540d09c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.878+0000 7f1d0cf43640 1 -- 172.21.15.16:0/3324622592 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d00037070 con 0x7f1d0815dfa0 2024-04-24T23:49:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.878+0000 7f1d0cf43640 1 -- 172.21.15.16:0/3324622592 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d00032440 con 0x7f1d0815dfa0 2024-04-24T23:49:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.878+0000 7f1d0cf43640 1 -- 172.21.15.16:0/3324622592 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d00032810 con 0x7f1d0815dfa0 2024-04-24T23:49:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.879+0000 7f1d101a9640 1 -- 172.21.15.16:0/3324622592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d0815dfa0 msgr2=0x7f1d080d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.879+0000 7f1d101a9640 1 --2- 172.21.15.16:0/3324622592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d0815dfa0 0x7f1d080d0fe0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1d080cf3e0 tx=0x7f1d0002a210 comp rx=0 tx=0).stop 2024-04-24T23:49:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.879+0000 7f1d101a9640 1 -- 172.21.15.16:0/3324622592 shutdown_connections 2024-04-24T23:49:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.879+0000 7f1d101a9640 1 --2- 172.21.15.16:0/3324622592 >> v2:172.21.15.156:3300/0 conn(0x7f1d080cdae0 0x7f1d080cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.879+0000 7f1d101a9640 1 --2- 172.21.15.16:0/3324622592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d0815dfa0 0x7f1d080d0fe0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.879+0000 7f1d101a9640 1 -- 172.21.15.16:0/3324622592 >> 172.21.15.16:0/3324622592 conn(0x7f1d0806b4d0 msgr2=0x7f1d080b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.879+0000 7f1d101a9640 1 -- 172.21.15.16:0/3324622592 shutdown_connections 2024-04-24T23:49:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.879+0000 7f1d101a9640 1 -- 172.21.15.16:0/3324622592 wait complete. 2024-04-24T23:49:57.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.880+0000 7f1d101a9640 1 Processor -- start 2024-04-24T23:49:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.880+0000 7f1d101a9640 1 -- start start 2024-04-24T23:49:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d101a9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d080cdae0 0x7f1d08146200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d101a9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d080ce5f0 0x7f1d08146740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d101a9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d0815dfa0 0x7f1d081541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d101a9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1d0815e380 con 0x7f1d080cdae0 2024-04-24T23:49:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d101a9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1d080cc2d0 con 0x7f1d080ce5f0 2024-04-24T23:49:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d101a9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1d080d1520 con 0x7f1d0815dfa0 2024-04-24T23:49:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d0df45640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d080cdae0 0x7f1d08146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d0df45640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d080cdae0 0x7f1d08146200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56618/0 (socket says 172.21.15.16:56618) 2024-04-24T23:49:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d0df45640 1 -- 172.21.15.16:0/2692192039 learned_addr learned my addr 172.21.15.16:0/2692192039 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:57.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d0d744640 1 --2- 172.21.15.16:0/2692192039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d080ce5f0 0x7f1d08146740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d0e746640 1 --2- 172.21.15.16:0/2692192039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d0815dfa0 0x7f1d081541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d0df45640 1 -- 172.21.15.16:0/2692192039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d0815dfa0 msgr2=0x7f1d081541c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d0df45640 1 --2- 172.21.15.16:0/2692192039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d0815dfa0 0x7f1d081541c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d0df45640 1 -- 172.21.15.16:0/2692192039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d080ce5f0 msgr2=0x7f1d08146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d0df45640 1 --2- 172.21.15.16:0/2692192039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d080ce5f0 0x7f1d08146740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.881+0000 7f1d0df45640 1 -- 172.21.15.16:0/2692192039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d081551f0 con 0x7f1d080cdae0 2024-04-24T23:49:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.882+0000 7f1d0df45640 1 --2- 172.21.15.16:0/2692192039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d080cdae0 0x7f1d08146200 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1d0002a250 tx=0x7f1d000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.882+0000 7f1d067fc640 1 -- 172.21.15.16:0/2692192039 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d00041070 con 0x7f1d080cdae0 2024-04-24T23:49:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.882+0000 7f1d067fc640 1 -- 172.21.15.16:0/2692192039 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1d0003c890 con 0x7f1d080cdae0 2024-04-24T23:49:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.882+0000 7f1d067fc640 1 -- 172.21.15.16:0/2692192039 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d00037040 con 0x7f1d080cdae0 2024-04-24T23:49:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.882+0000 7f1d101a9640 1 -- 172.21.15.16:0/2692192039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d081554d0 con 0x7f1d080cdae0 2024-04-24T23:49:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.882+0000 7f1d101a9640 1 -- 172.21.15.16:0/2692192039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d081576b0 con 0x7f1d080cdae0 2024-04-24T23:49:57.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.883+0000 7f1d101a9640 1 -- 172.21.15.16:0/2692192039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d081559f0 con 0x7f1d080cdae0 2024-04-24T23:49:57.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.883+0000 7f1d067fc640 1 -- 172.21.15.16:0/2692192039 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1d00044070 con 0x7f1d080cdae0 2024-04-24T23:49:57.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.884+0000 7f1d067fc640 1 --2- 172.21.15.16:0/2692192039 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1cf406be30 0x7f1cf406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:57.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.884+0000 7f1d067fc640 1 -- 172.21.15.16:0/2692192039 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f1d000ac1f0 con 0x7f1d080cdae0 2024-04-24T23:49:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.884+0000 7f1d0d744640 1 --2- 172.21.15.16:0/2692192039 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1cf406be30 0x7f1cf406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:57.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.885+0000 7f1d0d744640 1 --2- 172.21.15.16:0/2692192039 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1cf406be30 0x7f1cf406e2f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc007fd0 tx=0x7f1cfc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:57.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:57.888+0000 7f1d067fc640 1 -- 172.21.15.16:0/2692192039 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1d081559f0 con 0x7f1d080cdae0 2024-04-24T23:49:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.086+0000 7f1d101a9640 1 -- 172.21.15.16:0/2692192039 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1d0806bea0 con 0x7f1d080cdae0 2024-04-24T23:49:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.086+0000 7f1d067fc640 1 -- 172.21.15.16:0/2692192039 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f1d0007a8b0 con 0x7f1d080cdae0 2024-04-24T23:49:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-04-24T23:49:58.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.088+0000 7f1d101a9640 1 -- 172.21.15.16:0/2692192039 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1cf406be30 msgr2=0x7f1cf406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.088+0000 7f1d101a9640 1 --2- 172.21.15.16:0/2692192039 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1cf406be30 0x7f1cf406e2f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc007fd0 tx=0x7f1cfc009040 comp rx=0 tx=0).stop 2024-04-24T23:49:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.088+0000 7f1d101a9640 1 -- 172.21.15.16:0/2692192039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d080cdae0 msgr2=0x7f1d08146200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.088+0000 7f1d101a9640 1 --2- 172.21.15.16:0/2692192039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d080cdae0 0x7f1d08146200 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f1d0002a250 tx=0x7f1d000046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.089+0000 7f1d101a9640 1 -- 172.21.15.16:0/2692192039 shutdown_connections 2024-04-24T23:49:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.089+0000 7f1d101a9640 1 --2- 172.21.15.16:0/2692192039 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1cf406be30 0x7f1cf406e2f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.089+0000 7f1d101a9640 1 --2- 172.21.15.16:0/2692192039 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1d0815dfa0 0x7f1d081541c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.089+0000 7f1d101a9640 1 --2- 172.21.15.16:0/2692192039 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1d080ce5f0 0x7f1d08146740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.089+0000 7f1d101a9640 1 --2- 172.21.15.16:0/2692192039 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1d080cdae0 0x7f1d08146200 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.089+0000 7f1d101a9640 1 -- 172.21.15.16:0/2692192039 >> 172.21.15.16:0/2692192039 conn(0x7f1d0806b4d0 msgr2=0x7f1d080b5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.089+0000 7f1d101a9640 1 -- 172.21.15.16:0/2692192039 shutdown_connections 2024-04-24T23:49:58.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.089+0000 7f1d101a9640 1 -- 172.21.15.16:0/2692192039 wait complete. 2024-04-24T23:49:58.099 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:49:58.099 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:49:58.118 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:49:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.265+0000 7ffa152a5640 1 Processor -- start 2024-04-24T23:49:58.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.265+0000 7ffa152a5640 1 -- start start 2024-04-24T23:49:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.266+0000 7ffa152a5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa1015dfa0 0x7ffa100d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.266+0000 7ffa152a5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffa100cdae0 0x7ffa100cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.266+0000 7ffa152a5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa100d05a0 con 0x7ffa100ce5f0 2024-04-24T23:49:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.266+0000 7ffa152a5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ffa100b7360 con 0x7ffa100cdae0 2024-04-24T23:49:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.266+0000 7ffa152a5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ffa100cc640 con 0x7ffa1015dfa0 2024-04-24T23:49:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.266+0000 7ffa0f7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffa100cdae0 0x7ffa100cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.266+0000 7ffa0e7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffa100ce5f0 0x7ffa100b5f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58898/0 (socket says 172.21.15.16:58898) 2024-04-24T23:49:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.266+0000 7ffa0f7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffa100cdae0 0x7ffa100cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56620/0 (socket says 172.21.15.16:56620) 2024-04-24T23:49:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.266+0000 7ffa0e7fc640 1 -- 172.21.15.16:0/1559944074 learned_addr learned my addr 172.21.15.16:0/1559944074 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.266+0000 7ffa0effd640 1 --2- 172.21.15.16:0/1559944074 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa1015dfa0 0x7ffa100d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.267+0000 7ffa0f7fe640 1 -- 172.21.15.16:0/1559944074 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa1015dfa0 msgr2=0x7ffa100d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:58.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.267+0000 7ffa0f7fe640 1 --2- 172.21.15.16:0/1559944074 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa1015dfa0 0x7ffa100d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.267+0000 7ffa0f7fe640 1 -- 172.21.15.16:0/1559944074 >> v1:172.21.15.16:6789/0 conn(0x7ffa100ce5f0 legacy=0x7ffa100b5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.267+0000 7ffa0f7fe640 1 -- 172.21.15.16:0/1559944074 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa100d0880 con 0x7ffa100cdae0 2024-04-24T23:49:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.267+0000 7ffa0dffb640 1 -- 172.21.15.16:0/1559944074 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1307718435 0 0) 0x7ffa100d05a0 con 0x7ffa100ce5f0 2024-04-24T23:49:58.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.267+0000 7ffa0effd640 1 --2- 172.21.15.16:0/1559944074 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa1015dfa0 0x7ffa100d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.267+0000 7ffa0f7fe640 1 --2- 172.21.15.16:0/1559944074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa100cdae0 0x7ffa100cea70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ffa100cc7c0 tx=0x7ffa0002a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db7f2b4105e8356d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.267+0000 7ffa0dffb640 1 -- 172.21.15.16:0/1559944074 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0002ae60 con 0x7ffa100cdae0 2024-04-24T23:49:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.267+0000 7ffa0dffb640 1 -- 172.21.15.16:0/1559944074 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa00007740 con 0x7ffa100cdae0 2024-04-24T23:49:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.267+0000 7ffa0dffb640 1 -- 172.21.15.16:0/1559944074 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa00007b10 con 0x7ffa100cdae0 2024-04-24T23:49:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.268+0000 7ffa152a5640 1 -- 172.21.15.16:0/1559944074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa100cdae0 msgr2=0x7ffa100cea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.268+0000 7ffa152a5640 1 --2- 172.21.15.16:0/1559944074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa100cdae0 0x7ffa100cea70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7ffa100cc7c0 tx=0x7ffa0002a640 comp rx=0 tx=0).stop 2024-04-24T23:49:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.268+0000 7ffa152a5640 1 -- 172.21.15.16:0/1559944074 shutdown_connections 2024-04-24T23:49:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.268+0000 7ffa152a5640 1 --2- 172.21.15.16:0/1559944074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa100cdae0 0x7ffa100cea70 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.268+0000 7ffa152a5640 1 --2- 172.21.15.16:0/1559944074 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa1015dfa0 0x7ffa100d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.268+0000 7ffa152a5640 1 -- 172.21.15.16:0/1559944074 >> 172.21.15.16:0/1559944074 conn(0x7ffa1006b4d0 msgr2=0x7ffa100b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.268+0000 7ffa152a5640 1 -- 172.21.15.16:0/1559944074 shutdown_connections 2024-04-24T23:49:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.268+0000 7ffa152a5640 1 -- 172.21.15.16:0/1559944074 wait complete. 2024-04-24T23:49:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.269+0000 7ffa152a5640 1 Processor -- start 2024-04-24T23:49:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.269+0000 7ffa152a5640 1 -- start start 2024-04-24T23:49:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.270+0000 7ffa152a5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa100cdae0 0x7ffa10152c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.270+0000 7ffa152a5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa100ce5f0 0x7ffa10158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.270+0000 7ffa152a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa1015dfa0 0x7ffa10156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.270+0000 7ffa152a5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1015e380 con 0x7ffa1015dfa0 2024-04-24T23:49:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.270+0000 7ffa152a5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ffa100cc2d0 con 0x7ffa100ce5f0 2024-04-24T23:49:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.270+0000 7ffa152a5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ffa100d1520 con 0x7ffa100cdae0 2024-04-24T23:49:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.270+0000 7ffa0f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa1015dfa0 0x7ffa10156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.270+0000 7ffa0f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa1015dfa0 0x7ffa10156890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56622/0 (socket says 172.21.15.16:56622) 2024-04-24T23:49:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.270+0000 7ffa0f7fe640 1 -- 172.21.15.16:0/2902894308 learned_addr learned my addr 172.21.15.16:0/2902894308 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.270+0000 7ffa0e7fc640 1 --2- 172.21.15.16:0/2902894308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa100ce5f0 0x7ffa10158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.270+0000 7ffa0effd640 1 --2- 172.21.15.16:0/2902894308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa100cdae0 0x7ffa10152c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.272 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.271+0000 7ffa0f7fe640 1 -- 172.21.15.16:0/2902894308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa100cdae0 msgr2=0x7ffa10152c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.271+0000 7ffa0f7fe640 1 --2- 172.21.15.16:0/2902894308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa100cdae0 0x7ffa10152c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.271+0000 7ffa0f7fe640 1 -- 172.21.15.16:0/2902894308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa100ce5f0 msgr2=0x7ffa10158750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.271+0000 7ffa0f7fe640 1 --2- 172.21.15.16:0/2902894308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa100ce5f0 0x7ffa10158750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.271+0000 7ffa0f7fe640 1 -- 172.21.15.16:0/2902894308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa10159050 con 0x7ffa1015dfa0 2024-04-24T23:49:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.271+0000 7ffa0f7fe640 1 --2- 172.21.15.16:0/2902894308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa1015dfa0 0x7ffa10156890 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ffa0002ac30 tx=0x7ffa00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.271+0000 7ffa077fe640 1 -- 172.21.15.16:0/2902894308 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa00040070 con 0x7ffa1015dfa0 2024-04-24T23:49:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.271+0000 7ffa077fe640 1 -- 172.21.15.16:0/2902894308 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa0002ae60 con 0x7ffa1015dfa0 2024-04-24T23:49:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.271+0000 7ffa152a5640 1 -- 172.21.15.16:0/2902894308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa10159330 con 0x7ffa1015dfa0 2024-04-24T23:49:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.271+0000 7ffa077fe640 1 -- 172.21.15.16:0/2902894308 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa00045460 con 0x7ffa1015dfa0 2024-04-24T23:49:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.271+0000 7ffa152a5640 1 -- 172.21.15.16:0/2902894308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa101598f0 con 0x7ffa1015dfa0 2024-04-24T23:49:58.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.272+0000 7ffa152a5640 1 -- 172.21.15.16:0/2902894308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa1006bea0 con 0x7ffa1015dfa0 2024-04-24T23:49:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.272+0000 7ffa077fe640 1 -- 172.21.15.16:0/2902894308 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ffa00039040 con 0x7ffa1015dfa0 2024-04-24T23:49:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.273+0000 7ffa077fe640 1 --2- 172.21.15.16:0/2902894308 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff9f006be30 0x7ff9f006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.273+0000 7ffa077fe640 1 -- 172.21.15.16:0/2902894308 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ffa000aca80 con 0x7ffa1015dfa0 2024-04-24T23:49:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.273+0000 7ffa0effd640 1 --2- 172.21.15.16:0/2902894308 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff9f006be30 0x7ff9f006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.273+0000 7ffa0effd640 1 --2- 172.21.15.16:0/2902894308 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff9f006be30 0x7ff9f006e2f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800a950 tx=0x7ffa08008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:58.276 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.277+0000 7ffa077fe640 1 -- 172.21.15.16:0/2902894308 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ffa0007b0f0 con 0x7ffa1015dfa0 2024-04-24T23:49:58.464 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.463+0000 7ffa152a5640 1 -- 172.21.15.16:0/2902894308 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ffa10159e10 con 0x7ffa1015dfa0 2024-04-24T23:49:58.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.465+0000 7ffa077fe640 1 -- 172.21.15.16:0/2902894308 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7ffa0007a9f0 con 0x7ffa1015dfa0 2024-04-24T23:49:58.465 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:49:58.465 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:49:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.467+0000 7ffa152a5640 1 -- 172.21.15.16:0/2902894308 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff9f006be30 msgr2=0x7ff9f006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.467+0000 7ffa152a5640 1 --2- 172.21.15.16:0/2902894308 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff9f006be30 0x7ff9f006e2f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffa0800a950 tx=0x7ffa08008040 comp rx=0 tx=0).stop 2024-04-24T23:49:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.467+0000 7ffa152a5640 1 -- 172.21.15.16:0/2902894308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa1015dfa0 msgr2=0x7ffa10156890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.467+0000 7ffa152a5640 1 --2- 172.21.15.16:0/2902894308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa1015dfa0 0x7ffa10156890 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ffa0002ac30 tx=0x7ffa00004a40 comp rx=0 tx=0).stop 2024-04-24T23:49:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.467+0000 7ffa152a5640 1 -- 172.21.15.16:0/2902894308 shutdown_connections 2024-04-24T23:49:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.467+0000 7ffa152a5640 1 --2- 172.21.15.16:0/2902894308 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff9f006be30 0x7ff9f006e2f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.467+0000 7ffa152a5640 1 --2- 172.21.15.16:0/2902894308 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa1015dfa0 0x7ffa10156890 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.467+0000 7ffa152a5640 1 --2- 172.21.15.16:0/2902894308 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa100ce5f0 0x7ffa10158750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.467+0000 7ffa152a5640 1 --2- 172.21.15.16:0/2902894308 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa100cdae0 0x7ffa10152c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.467+0000 7ffa152a5640 1 -- 172.21.15.16:0/2902894308 >> 172.21.15.16:0/2902894308 conn(0x7ffa1006b4d0 msgr2=0x7ffa10160890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:58.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.467+0000 7ffa152a5640 1 -- 172.21.15.16:0/2902894308 shutdown_connections 2024-04-24T23:49:58.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.467+0000 7ffa152a5640 1 -- 172.21.15.16:0/2902894308 wait complete. 2024-04-24T23:49:58.478 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:49:58.478 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:49:58.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.584+0000 7fa9b926d640 1 Processor -- start 2024-04-24T23:49:58.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.584+0000 7fa9b926d640 1 -- start start 2024-04-24T23:49:58.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.585+0000 7fa9b926d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa9b40d12d0 0x7fa9b40d16b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.585+0000 7fa9b926d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa9b415e5c0 0x7fa9b40d2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.585+0000 7fa9b926d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9b415ec30 con 0x7fa9b40cd790 2024-04-24T23:49:58.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.585+0000 7fa9b926d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa9b415f4d0 con 0x7fa9b415e5c0 2024-04-24T23:49:58.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.585+0000 7fa9b926d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa9b415ef10 con 0x7fa9b40d12d0 2024-04-24T23:49:58.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.585+0000 7fa9b37fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa9b415e5c0 0x7fa9b40d2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.585+0000 7fa9abfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa9b40cd790 0x7fa9b40c8580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58908/0 (socket says 172.21.15.16:58908) 2024-04-24T23:49:58.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.585+0000 7fa9abfff640 1 -- 172.21.15.16:0/2392530848 learned_addr learned my addr 172.21.15.16:0/2392530848 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:58.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.585+0000 7fa9b2ffd640 1 --2- 172.21.15.16:0/2392530848 >> v2:172.21.15.156:3300/0 conn(0x7fa9b40d12d0 0x7fa9b40d16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.585+0000 7fa9b37fe640 1 -- 172.21.15.16:0/2392530848 >> v2:172.21.15.156:3300/0 conn(0x7fa9b40d12d0 msgr2=0x7fa9b40d16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:58.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.585+0000 7fa9b37fe640 1 --2- 172.21.15.16:0/2392530848 >> v2:172.21.15.156:3300/0 conn(0x7fa9b40d12d0 0x7fa9b40d16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.585+0000 7fa9b37fe640 1 -- 172.21.15.16:0/2392530848 >> v1:172.21.15.16:6789/0 conn(0x7fa9b40cd790 legacy=0x7fa9b40c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.585+0000 7fa9b37fe640 1 -- 172.21.15.16:0/2392530848 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b415f1f0 con 0x7fa9b415e5c0 2024-04-24T23:49:58.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.585+0000 7fa9b27fc640 1 -- 172.21.15.16:0/2392530848 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2219992380 0 0) 0x7fa9b415ec30 con 0x7fa9b40cd790 2024-04-24T23:49:58.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.586+0000 7fa9b37fe640 1 --2- 172.21.15.16:0/2392530848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9b415e5c0 0x7fa9b40d2130 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40c7600 tx=0x7fa9ac02a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff7623afcc6d390d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:58.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.586+0000 7fa9b27fc640 1 -- 172.21.15.16:0/2392530848 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac007740 con 0x7fa9b415e5c0 2024-04-24T23:49:58.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.586+0000 7fa9b27fc640 1 -- 172.21.15.16:0/2392530848 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9ac0078e0 con 0x7fa9b415e5c0 2024-04-24T23:49:58.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.586+0000 7fa9b27fc640 1 -- 172.21.15.16:0/2392530848 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ac007cb0 con 0x7fa9b415e5c0 2024-04-24T23:49:58.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.587+0000 7fa9b926d640 1 -- 172.21.15.16:0/2392530848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9b415e5c0 msgr2=0x7fa9b40d2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.587+0000 7fa9b926d640 1 --2- 172.21.15.16:0/2392530848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9b415e5c0 0x7fa9b40d2130 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa9b40c7600 tx=0x7fa9ac02a560 comp rx=0 tx=0).stop 2024-04-24T23:49:58.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.587+0000 7fa9b926d640 1 -- 172.21.15.16:0/2392530848 shutdown_connections 2024-04-24T23:49:58.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.587+0000 7fa9b926d640 1 --2- 172.21.15.16:0/2392530848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9b415e5c0 0x7fa9b40d2130 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.587+0000 7fa9b926d640 1 --2- 172.21.15.16:0/2392530848 >> v2:172.21.15.156:3300/0 conn(0x7fa9b40d12d0 0x7fa9b40d16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.587+0000 7fa9b926d640 1 -- 172.21.15.16:0/2392530848 >> 172.21.15.16:0/2392530848 conn(0x7fa9b4072e70 msgr2=0x7fa9b4073250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:58.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.587+0000 7fa9b926d640 1 -- 172.21.15.16:0/2392530848 shutdown_connections 2024-04-24T23:49:58.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.587+0000 7fa9b926d640 1 -- 172.21.15.16:0/2392530848 wait complete. 2024-04-24T23:49:58.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.588+0000 7fa9b926d640 1 Processor -- start 2024-04-24T23:49:58.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.588+0000 7fa9b926d640 1 -- start start 2024-04-24T23:49:58.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b926d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9b40cd790 0x7fa9b40b02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b926d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b40d12d0 0x7fa9b41572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b926d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9b415e5c0 0x7fa9b4157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b926d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b40d27f0 con 0x7fa9b40cd790 2024-04-24T23:49:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b926d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa9b40d2670 con 0x7fa9b40d12d0 2024-04-24T23:49:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b926d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fa9b40d2970 con 0x7fa9b415e5c0 2024-04-24T23:49:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9b40cd790 0x7fa9b40b02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9b40cd790 0x7fa9b40b02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56648/0 (socket says 172.21.15.16:56648) 2024-04-24T23:49:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b2ffd640 1 -- 172.21.15.16:0/1569733541 learned_addr learned my addr 172.21.15.16:0/1569733541 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9abfff640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b40d12d0 0x7fa9b41572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b37fe640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9b415e5c0 0x7fa9b4157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b2ffd640 1 -- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9b415e5c0 msgr2=0x7fa9b4157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b2ffd640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9b415e5c0 0x7fa9b4157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b2ffd640 1 -- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b40d12d0 msgr2=0x7fa9b41572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b2ffd640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b40d12d0 0x7fa9b41572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b2ffd640 1 -- 172.21.15.16:0/1569733541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9b41569a0 con 0x7fa9b40cd790 2024-04-24T23:49:58.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b37fe640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9b415e5c0 0x7fa9b4157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:58.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.589+0000 7fa9b2ffd640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9b40cd790 0x7fa9b40b02a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400d800 tx=0x7fa9a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:58.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.590+0000 7fa9ab7fe640 1 -- 172.21.15.16:0/1569733541 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a400b6c0 con 0x7fa9b40cd790 2024-04-24T23:49:58.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.590+0000 7fa9ab7fe640 1 -- 172.21.15.16:0/1569733541 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9a4019070 con 0x7fa9b40cd790 2024-04-24T23:49:58.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.590+0000 7fa9b926d640 1 -- 172.21.15.16:0/1569733541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9b40aeb50 con 0x7fa9b40cd790 2024-04-24T23:49:58.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.590+0000 7fa9ab7fe640 1 -- 172.21.15.16:0/1569733541 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a400ba80 con 0x7fa9b40cd790 2024-04-24T23:49:58.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.590+0000 7fa9b926d640 1 -- 172.21.15.16:0/1569733541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9b41563e0 con 0x7fa9b40cd790 2024-04-24T23:49:58.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.590+0000 7fa9b926d640 1 -- 172.21.15.16:0/1569733541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9b40cc510 con 0x7fa9b40cd790 2024-04-24T23:49:58.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.591+0000 7fa9ab7fe640 1 -- 172.21.15.16:0/1569733541 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa9a400bc20 con 0x7fa9b40cd790 2024-04-24T23:49:58.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.591+0000 7fa9ab7fe640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa99406be30 0x7fa99406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.591+0000 7fa9ab7fe640 1 -- 172.21.15.16:0/1569733541 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa9a408b730 con 0x7fa9b40cd790 2024-04-24T23:49:58.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.592+0000 7fa9abfff640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa99406be30 0x7fa99406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.592+0000 7fa9abfff640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa99406be30 0x7fa99406e2f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4158670 tx=0x7fa99c007b70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:58.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.596+0000 7fa9ab7fe640 1 -- 172.21.15.16:0/1569733541 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa9b40cc510 con 0x7fa9b40cd790 2024-04-24T23:49:58.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.777+0000 7fa9b926d640 1 -- 172.21.15.16:0/1569733541 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa9b4068f90 con 0x7fa9b40cd790 2024-04-24T23:49:58.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.778+0000 7fa9ab7fe640 1 -- 172.21.15.16:0/1569733541 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7fa9a401c360 con 0x7fa9b40cd790 2024-04-24T23:49:58.778 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:49:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.780+0000 7fa9b926d640 1 -- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa99406be30 msgr2=0x7fa99406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.780+0000 7fa9b926d640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa99406be30 0x7fa99406e2f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa9b4158670 tx=0x7fa99c007b70 comp rx=0 tx=0).stop 2024-04-24T23:49:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.780+0000 7fa9b926d640 1 -- 172.21.15.16:0/1569733541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9b40cd790 msgr2=0x7fa9b40b02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.780+0000 7fa9b926d640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9b40cd790 0x7fa9b40b02a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fa9a400d800 tx=0x7fa9a400dcc0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.780+0000 7fa9b926d640 1 -- 172.21.15.16:0/1569733541 shutdown_connections 2024-04-24T23:49:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.780+0000 7fa9b926d640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa99406be30 0x7fa99406e2f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.780+0000 7fa9b926d640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa9b415e5c0 0x7fa9b4157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.780+0000 7fa9b926d640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa9b40d12d0 0x7fa9b41572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.780+0000 7fa9b926d640 1 --2- 172.21.15.16:0/1569733541 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9b40cd790 0x7fa9b40b02a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.780+0000 7fa9b926d640 1 -- 172.21.15.16:0/1569733541 >> 172.21.15.16:0/1569733541 conn(0x7fa9b4072e70 msgr2=0x7fa9b40ce390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.780+0000 7fa9b926d640 1 -- 172.21.15.16:0/1569733541 shutdown_connections 2024-04-24T23:49:58.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.780+0000 7fa9b926d640 1 -- 172.21.15.16:0/1569733541 wait complete. 2024-04-24T23:49:58.791 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:49:58.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.896+0000 7f3ab3264640 1 Processor -- start 2024-04-24T23:49:58.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.896+0000 7f3ab3264640 1 -- start start 2024-04-24T23:49:58.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.896+0000 7f3ab3264640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3aac15dfd0 0x7f3aac06b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.896+0000 7f3ab3264640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3aac06bc40 0x7f3aac06c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.897+0000 7f3ab3264640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aac0d24d0 con 0x7f3aac0cefc0 2024-04-24T23:49:58.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.897+0000 7f3ab3264640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f3aac0d27b0 con 0x7f3aac15dfd0 2024-04-24T23:49:58.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.897+0000 7f3ab3264640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f3aac0d2a90 con 0x7f3aac06bc40 2024-04-24T23:49:58.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.897+0000 7f3ab1000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3aac15dfd0 0x7f3aac06b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.897+0000 7f3ab1000640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3aac15dfd0 0x7f3aac06b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56656/0 (socket says 172.21.15.16:56656) 2024-04-24T23:49:58.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.897+0000 7f3ab1000640 1 -- 172.21.15.16:0/1228983641 learned_addr learned my addr 172.21.15.16:0/1228983641 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.897+0000 7f3aabfff640 1 --2- 172.21.15.16:0/1228983641 >> v2:172.21.15.156:3300/0 conn(0x7f3aac06bc40 0x7f3aac06c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.897+0000 7f3ab1000640 1 -- 172.21.15.16:0/1228983641 >> v2:172.21.15.156:3300/0 conn(0x7f3aac06bc40 msgr2=0x7f3aac06c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.897+0000 7f3ab1000640 1 --2- 172.21.15.16:0/1228983641 >> v2:172.21.15.156:3300/0 conn(0x7f3aac06bc40 0x7f3aac06c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.897+0000 7f3ab1000640 1 -- 172.21.15.16:0/1228983641 >> v1:172.21.15.156:6789/0 conn(0x7f3aac0cefc0 legacy=0x7f3aac0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.897+0000 7f3ab1000640 1 -- 172.21.15.16:0/1228983641 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aac0d2d70 con 0x7f3aac15dfd0 2024-04-24T23:49:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.898+0000 7f3ab1000640 1 --2- 172.21.15.16:0/1228983641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac15dfd0 0x7f3aac06b2f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3aac06cf60 tx=0x7f3a9c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d916010482e507f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.898+0000 7f3aab7fe640 1 -- 172.21.15.16:0/1228983641 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c037070 con 0x7f3aac15dfd0 2024-04-24T23:49:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.898+0000 7f3aab7fe640 1 -- 172.21.15.16:0/1228983641 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a9c032440 con 0x7f3aac15dfd0 2024-04-24T23:49:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.898+0000 7f3aab7fe640 1 -- 172.21.15.16:0/1228983641 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c032810 con 0x7f3aac15dfd0 2024-04-24T23:49:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.899+0000 7f3ab3264640 1 -- 172.21.15.16:0/1228983641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac15dfd0 msgr2=0x7f3aac06b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.899+0000 7f3ab3264640 1 --2- 172.21.15.16:0/1228983641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac15dfd0 0x7f3aac06b2f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3aac06cf60 tx=0x7f3a9c02a1a0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.899+0000 7f3ab3264640 1 -- 172.21.15.16:0/1228983641 shutdown_connections 2024-04-24T23:49:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.899+0000 7f3ab3264640 1 --2- 172.21.15.16:0/1228983641 >> v2:172.21.15.156:3300/0 conn(0x7f3aac06bc40 0x7f3aac06c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.899+0000 7f3ab3264640 1 --2- 172.21.15.16:0/1228983641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac15dfd0 0x7f3aac06b2f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.899+0000 7f3ab3264640 1 -- 172.21.15.16:0/1228983641 >> 172.21.15.16:0/1228983641 conn(0x7f3aac0d3dc0 msgr2=0x7f3aac0d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.899+0000 7f3ab3264640 1 -- 172.21.15.16:0/1228983641 shutdown_connections 2024-04-24T23:49:58.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.899+0000 7f3ab3264640 1 -- 172.21.15.16:0/1228983641 wait complete. 2024-04-24T23:49:58.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.900+0000 7f3ab3264640 1 Processor -- start 2024-04-24T23:49:58.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.900+0000 7f3ab3264640 1 -- start start 2024-04-24T23:49:58.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab3264640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac06bc40 0x7f3aac15a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab3264640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac0cefc0 0x7f3aac15f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab3264640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aac15dfd0 0x7f3aac15f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab3264640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac06b830 con 0x7f3aac06bc40 2024-04-24T23:49:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab3264640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3aac0cead0 con 0x7f3aac15dfd0 2024-04-24T23:49:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab3264640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f3aac06b9b0 con 0x7f3aac0cefc0 2024-04-24T23:49:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab1000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac06bc40 0x7f3aac15a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab1000640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac06bc40 0x7f3aac15a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56664/0 (socket says 172.21.15.16:56664) 2024-04-24T23:49:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab1000640 1 -- 172.21.15.16:0/1114585683 learned_addr learned my addr 172.21.15.16:0/1114585683 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3aabfff640 1 --2- 172.21.15.16:0/1114585683 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac0cefc0 0x7f3aac15f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab1801640 1 --2- 172.21.15.16:0/1114585683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aac15dfd0 0x7f3aac15f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab1000640 1 -- 172.21.15.16:0/1114585683 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac0cefc0 msgr2=0x7f3aac15f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab1000640 1 --2- 172.21.15.16:0/1114585683 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac0cefc0 0x7f3aac15f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab1000640 1 -- 172.21.15.16:0/1114585683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aac15dfd0 msgr2=0x7f3aac15f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab1000640 1 --2- 172.21.15.16:0/1114585683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aac15dfd0 0x7f3aac15f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab1000640 1 -- 172.21.15.16:0/1114585683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aac0cafc0 con 0x7f3aac06bc40 2024-04-24T23:49:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.901+0000 7f3ab1000640 1 --2- 172.21.15.16:0/1114585683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac06bc40 0x7f3aac15a670 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c02a1e0 tx=0x7f3a9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.902+0000 7f3aa97fa640 1 -- 172.21.15.16:0/1114585683 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c037040 con 0x7f3aac06bc40 2024-04-24T23:49:58.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.902+0000 7f3aa97fa640 1 -- 172.21.15.16:0/1114585683 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a9c042070 con 0x7f3aac06bc40 2024-04-24T23:49:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.902+0000 7f3ab3264640 1 -- 172.21.15.16:0/1114585683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aac0cb2a0 con 0x7f3aac06bc40 2024-04-24T23:49:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.902+0000 7f3aa97fa640 1 -- 172.21.15.16:0/1114585683 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c030bb0 con 0x7f3aac06bc40 2024-04-24T23:49:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.902+0000 7f3ab3264640 1 -- 172.21.15.16:0/1114585683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aac0cb860 con 0x7f3aac06bc40 2024-04-24T23:49:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.903+0000 7f3ab3264640 1 -- 172.21.15.16:0/1114585683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aac068f90 con 0x7f3aac06bc40 2024-04-24T23:49:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.903+0000 7f3aa97fa640 1 -- 172.21.15.16:0/1114585683 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3a9c047460 con 0x7f3aac06bc40 2024-04-24T23:49:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.904+0000 7f3aa97fa640 1 --2- 172.21.15.16:0/1114585683 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a9806be30 0x7f3a9806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.904+0000 7f3aa97fa640 1 -- 172.21.15.16:0/1114585683 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f3a9c0abe80 con 0x7f3aac06bc40 2024-04-24T23:49:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.904+0000 7f3aabfff640 1 --2- 172.21.15.16:0/1114585683 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a9806be30 0x7f3a9806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.904+0000 7f3aabfff640 1 --2- 172.21.15.16:0/1114585683 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a9806be30 0x7f3a9806e2f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0006fd0 tx=0x7f3aa0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:58.909 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:58.908+0000 7f3aa97fa640 1 -- 172.21.15.16:0/1114585683 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3a9c045360 con 0x7f3aac06bc40 2024-04-24T23:49:59.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.092+0000 7f3ab3264640 1 -- 172.21.15.16:0/1114585683 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3aac06cd30 con 0x7f3aac06bc40 2024-04-24T23:49:59.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.092+0000 7f3aa97fa640 1 -- 172.21.15.16:0/1114585683 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f3a9c07a4c0 con 0x7f3aac06bc40 2024-04-24T23:49:59.092 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:59.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.095+0000 7f3ab3264640 1 -- 172.21.15.16:0/1114585683 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a9806be30 msgr2=0x7f3a9806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.095+0000 7f3ab3264640 1 --2- 172.21.15.16:0/1114585683 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a9806be30 0x7f3a9806e2f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3aa0006fd0 tx=0x7f3aa0008040 comp rx=0 tx=0).stop 2024-04-24T23:49:59.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.095+0000 7f3ab3264640 1 -- 172.21.15.16:0/1114585683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac06bc40 msgr2=0x7f3aac15a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.095+0000 7f3ab3264640 1 --2- 172.21.15.16:0/1114585683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac06bc40 0x7f3aac15a670 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c02a1e0 tx=0x7f3a9c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.095+0000 7f3ab3264640 1 -- 172.21.15.16:0/1114585683 shutdown_connections 2024-04-24T23:49:59.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.095+0000 7f3ab3264640 1 --2- 172.21.15.16:0/1114585683 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a9806be30 0x7f3a9806e2f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.095+0000 7f3ab3264640 1 --2- 172.21.15.16:0/1114585683 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3aac15dfd0 0x7f3aac15f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.095+0000 7f3ab3264640 1 --2- 172.21.15.16:0/1114585683 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3aac0cefc0 0x7f3aac15f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.095+0000 7f3ab3264640 1 --2- 172.21.15.16:0/1114585683 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aac06bc40 0x7f3aac15a670 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.095+0000 7f3ab3264640 1 -- 172.21.15.16:0/1114585683 >> 172.21.15.16:0/1114585683 conn(0x7f3aac0d3dc0 msgr2=0x7f3aac06c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:59.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.095+0000 7f3ab3264640 1 -- 172.21.15.16:0/1114585683 shutdown_connections 2024-04-24T23:49:59.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.095+0000 7f3ab3264640 1 -- 172.21.15.16:0/1114585683 wait complete. 2024-04-24T23:49:59.105 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:49:59.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.210+0000 7f22e0f22640 1 Processor -- start 2024-04-24T23:49:59.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.210+0000 7f22e0f22640 1 -- start start 2024-04-24T23:49:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.211+0000 7f22e0f22640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f22dc15dfd0 0x7f22dc06b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.211+0000 7f22e0f22640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22dc06bc40 0x7f22dc06c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.211+0000 7f22e0f22640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22dc0d24d0 con 0x7f22dc0cefc0 2024-04-24T23:49:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.211+0000 7f22e0f22640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f22dc0d27b0 con 0x7f22dc06bc40 2024-04-24T23:49:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.211+0000 7f22e0f22640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f22dc0d2a90 con 0x7f22dc15dfd0 2024-04-24T23:49:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.211+0000 7f22d3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22dc06bc40 0x7f22dc06c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.211+0000 7f22d3fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22dc06bc40 0x7f22dc06c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56674/0 (socket says 172.21.15.16:56674) 2024-04-24T23:49:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.211+0000 7f22d3fff640 1 -- 172.21.15.16:0/1940248751 learned_addr learned my addr 172.21.15.16:0/1940248751 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.211+0000 7f22da59c640 1 --2- 172.21.15.16:0/1940248751 >> v2:172.21.15.156:3300/0 conn(0x7f22dc15dfd0 0x7f22dc06b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.211+0000 7f22d3fff640 1 -- 172.21.15.16:0/1940248751 >> v2:172.21.15.156:3300/0 conn(0x7f22dc15dfd0 msgr2=0x7f22dc06b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.211+0000 7f22d3fff640 1 --2- 172.21.15.16:0/1940248751 >> v2:172.21.15.156:3300/0 conn(0x7f22dc15dfd0 0x7f22dc06b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.211+0000 7f22d3fff640 1 -- 172.21.15.16:0/1940248751 >> v1:172.21.15.156:6789/0 conn(0x7f22dc0cefc0 legacy=0x7f22dc0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:59.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.211+0000 7f22d3fff640 1 -- 172.21.15.16:0/1940248751 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22dc0d2d70 con 0x7f22dc06bc40 2024-04-24T23:49:59.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.212+0000 7f22d3fff640 1 --2- 172.21.15.16:0/1940248751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22dc06bc40 0x7f22dc06c0d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f22dc0d3ce0 tx=0x7f22c402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5715e0c55e910f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:59.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.212+0000 7f22d9d9b640 1 -- 172.21.15.16:0/1940248751 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c4037070 con 0x7f22dc06bc40 2024-04-24T23:49:59.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.212+0000 7f22d9d9b640 1 -- 172.21.15.16:0/1940248751 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22c4032440 con 0x7f22dc06bc40 2024-04-24T23:49:59.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.212+0000 7f22d9d9b640 1 -- 172.21.15.16:0/1940248751 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c4032810 con 0x7f22dc06bc40 2024-04-24T23:49:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.213+0000 7f22e0f22640 1 -- 172.21.15.16:0/1940248751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22dc06bc40 msgr2=0x7f22dc06c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.213+0000 7f22e0f22640 1 --2- 172.21.15.16:0/1940248751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22dc06bc40 0x7f22dc06c0d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f22dc0d3ce0 tx=0x7f22c402a1a0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.213+0000 7f22e0f22640 1 -- 172.21.15.16:0/1940248751 shutdown_connections 2024-04-24T23:49:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.213+0000 7f22e0f22640 1 --2- 172.21.15.16:0/1940248751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22dc06bc40 0x7f22dc06c0d0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.213+0000 7f22e0f22640 1 --2- 172.21.15.16:0/1940248751 >> v2:172.21.15.156:3300/0 conn(0x7f22dc15dfd0 0x7f22dc06b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.213+0000 7f22e0f22640 1 -- 172.21.15.16:0/1940248751 >> 172.21.15.16:0/1940248751 conn(0x7f22dc0d3dc0 msgr2=0x7f22dc0d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.213+0000 7f22e0f22640 1 -- 172.21.15.16:0/1940248751 shutdown_connections 2024-04-24T23:49:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.213+0000 7f22e0f22640 1 -- 172.21.15.16:0/1940248751 wait complete. 2024-04-24T23:49:59.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.214+0000 7f22e0f22640 1 Processor -- start 2024-04-24T23:49:59.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.214+0000 7f22e0f22640 1 -- start start 2024-04-24T23:49:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22e0f22640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22dc06bc40 0x7f22dc1e6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22e0f22640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22dc0cefc0 0x7f22dc1e94b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22e0f22640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc15dfd0 0x7f22dc148bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22e0f22640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc06b830 con 0x7f22dc0cefc0 2024-04-24T23:49:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22e0f22640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc0cead0 con 0x7f22dc15dfd0 2024-04-24T23:49:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22e0f22640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f22dc06b9b0 con 0x7f22dc06bc40 2024-04-24T23:49:59.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22dc0cefc0 0x7f22dc1e94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22dc0cefc0 0x7f22dc1e94b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56680/0 (socket says 172.21.15.16:56680) 2024-04-24T23:49:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22d3fff640 1 -- 172.21.15.16:0/2799859048 learned_addr learned my addr 172.21.15.16:0/2799859048 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22da59c640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22dc06bc40 0x7f22dc1e6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22dad9d640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc15dfd0 0x7f22dc148bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22d3fff640 1 -- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22dc06bc40 msgr2=0x7f22dc1e6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22d3fff640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22dc06bc40 0x7f22dc1e6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22d3fff640 1 -- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc15dfd0 msgr2=0x7f22dc148bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22d3fff640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc15dfd0 0x7f22dc148bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22d3fff640 1 -- 172.21.15.16:0/2799859048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22dc1498e0 con 0x7f22dc0cefc0 2024-04-24T23:49:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22da59c640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22dc06bc40 0x7f22dc1e6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22dad9d640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc15dfd0 0x7f22dc148bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:49:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.215+0000 7f22d3fff640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22dc0cefc0 0x7f22dc1e94b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f22c4000c00 tx=0x7f22c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:59.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.216+0000 7f22d2ffd640 1 -- 172.21.15.16:0/2799859048 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c4041020 con 0x7f22dc0cefc0 2024-04-24T23:49:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.216+0000 7f22d2ffd640 1 -- 172.21.15.16:0/2799859048 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22c4030970 con 0x7f22dc0cefc0 2024-04-24T23:49:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.216+0000 7f22e0f22640 1 -- 172.21.15.16:0/2799859048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22dc149bc0 con 0x7f22dc0cefc0 2024-04-24T23:49:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.216+0000 7f22d2ffd640 1 -- 172.21.15.16:0/2799859048 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22c4037040 con 0x7f22dc0cefc0 2024-04-24T23:49:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.216+0000 7f22e0f22640 1 -- 172.21.15.16:0/2799859048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22dc14a180 con 0x7f22dc0cefc0 2024-04-24T23:49:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.217+0000 7f22e0f22640 1 -- 172.21.15.16:0/2799859048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22dc068f90 con 0x7f22dc0cefc0 2024-04-24T23:49:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.217+0000 7f22d2ffd640 1 -- 172.21.15.16:0/2799859048 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f22c4044020 con 0x7f22dc0cefc0 2024-04-24T23:49:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.217+0000 7f22d2ffd640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f22bc06be30 0x7f22bc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.218+0000 7f22d2ffd640 1 -- 172.21.15.16:0/2799859048 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f22c40abc40 con 0x7f22dc0cefc0 2024-04-24T23:49:59.219 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.218+0000 7f22da59c640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f22bc06be30 0x7f22bc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.220 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.218+0000 7f22da59c640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f22bc06be30 0x7f22bc06e2f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f22dc1e9210 tx=0x7f22cc008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:59.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.222+0000 7f22d2ffd640 1 -- 172.21.15.16:0/2799859048 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f22c404b030 con 0x7f22dc0cefc0 2024-04-24T23:49:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.405+0000 7f22e0f22640 1 -- 172.21.15.16:0/2799859048 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f22dc1e99f0 con 0x7f22dc0cefc0 2024-04-24T23:49:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.405+0000 7f22d2ffd640 1 -- 172.21.15.16:0/2799859048 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f22c407a1f0 con 0x7f22dc0cefc0 2024-04-24T23:49:59.405 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:49:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.407+0000 7f22e0f22640 1 -- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f22bc06be30 msgr2=0x7f22bc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.407+0000 7f22e0f22640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f22bc06be30 0x7f22bc06e2f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f22dc1e9210 tx=0x7f22cc008040 comp rx=0 tx=0).stop 2024-04-24T23:49:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.408+0000 7f22e0f22640 1 -- 172.21.15.16:0/2799859048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22dc0cefc0 msgr2=0x7f22dc1e94b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.408+0000 7f22e0f22640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22dc0cefc0 0x7f22dc1e94b0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f22c4000c00 tx=0x7f22c40046c0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.408+0000 7f22e0f22640 1 -- 172.21.15.16:0/2799859048 shutdown_connections 2024-04-24T23:49:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.408+0000 7f22e0f22640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f22bc06be30 0x7f22bc06e2f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.408+0000 7f22e0f22640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc15dfd0 0x7f22dc148bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.408+0000 7f22e0f22640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22dc0cefc0 0x7f22dc1e94b0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.408+0000 7f22e0f22640 1 --2- 172.21.15.16:0/2799859048 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f22dc06bc40 0x7f22dc1e6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.408+0000 7f22e0f22640 1 -- 172.21.15.16:0/2799859048 >> 172.21.15.16:0/2799859048 conn(0x7f22dc0d3dc0 msgr2=0x7f22dc0cf3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.408+0000 7f22e0f22640 1 -- 172.21.15.16:0/2799859048 shutdown_connections 2024-04-24T23:49:59.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.408+0000 7f22e0f22640 1 -- 172.21.15.16:0/2799859048 wait complete. 2024-04-24T23:49:59.418 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:49:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.524+0000 7f0d6b9ce640 1 Processor -- start 2024-04-24T23:49:59.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.525+0000 7f0d6b9ce640 1 -- start start 2024-04-24T23:49:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.525+0000 7f0d6b9ce640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d6406bc40 0x7f0d6406c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.525+0000 7f0d6b9ce640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d640d24d0 con 0x7f0d640b7050 2024-04-24T23:49:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.525+0000 7f0d6b9ce640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d640d27b0 con 0x7f0d6415dfd0 2024-04-24T23:49:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.525+0000 7f0d6b9ce640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0d6406c5e0 con 0x7f0d6406bc40 2024-04-24T23:49:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d69f6b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0d640b7050 0x7f0d640b74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58922/0 (socket says 172.21.15.16:58922) 2024-04-24T23:49:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d69f6b640 1 -- 172.21.15.16:0/1812967898 learned_addr learned my addr 172.21.15.16:0/1812967898 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:59.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d68f69640 1 --2- 172.21.15.16:0/1812967898 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d6406bc40 0x7f0d6406c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 -- 172.21.15.16:0/1812967898 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 864885092 0 0) 0x7f0d640d24d0 con 0x7f0d640b7050 2024-04-24T23:49:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 -- 172.21.15.16:0/1812967898 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d50003520 con 0x7f0d640b7050 2024-04-24T23:49:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 -- 172.21.15.16:0/1812967898 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 861390208 0 0) 0x7f0d50003520 con 0x7f0d640b7050 2024-04-24T23:49:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 -- 172.21.15.16:0/1812967898 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d6406bc40 msgr2=0x7f0d6406c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 --2- 172.21.15.16:0/1812967898 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d6406bc40 0x7f0d6406c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 -- 172.21.15.16:0/1812967898 >> v1:172.21.15.156:6789/0 conn(0x7f0d6415dfd0 legacy=0x7f0d6406b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 -- 172.21.15.16:0/1812967898 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d640d2a90 con 0x7f0d640b7050 2024-04-24T23:49:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 -- 172.21.15.16:0/1812967898 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f0d5c0019f0 con 0x7f0d640b7050 2024-04-24T23:49:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d68f69640 1 --2- 172.21.15.16:0/1812967898 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d6406bc40 0x7f0d6406c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 -- 172.21.15.16:0/1812967898 >> v1:172.21.15.16:6789/0 conn(0x7f0d640b7050 legacy=0x7f0d640b74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 --2- 172.21.15.16:0/1812967898 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d6406bc40 0x7f0d500039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 --2- 172.21.15.16:0/1812967898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50003fd0 0x7f0d50004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 --2- 172.21.15.16:0/1812967898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d50004d40 0x7f0d500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.529 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 -- 172.21.15.16:0/1812967898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5000bb80 con 0x7f0d50003fd0 2024-04-24T23:49:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 -- 172.21.15.16:0/1812967898 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0d5000bd30 con 0x7f0d50004d40 2024-04-24T23:49:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d5bfff640 1 -- 172.21.15.16:0/1812967898 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0d5000bf30 con 0x7f0d6406bc40 2024-04-24T23:49:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d68f69640 1 --2- 172.21.15.16:0/1812967898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50003fd0 0x7f0d50004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d6976a640 1 --2- 172.21.15.16:0/1812967898 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d6406bc40 0x7f0d500039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.526+0000 7f0d69f6b640 1 --2- 172.21.15.16:0/1812967898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d50004d40 0x7f0d500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.527+0000 7f0d68f69640 1 -- 172.21.15.16:0/1812967898 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d6406bc40 msgr2=0x7f0d500039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.527+0000 7f0d68f69640 1 --2- 172.21.15.16:0/1812967898 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d6406bc40 0x7f0d500039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.527+0000 7f0d68f69640 1 -- 172.21.15.16:0/1812967898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d50004d40 msgr2=0x7f0d500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.527+0000 7f0d68f69640 1 --2- 172.21.15.16:0/1812967898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d50004d40 0x7f0d500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.527+0000 7f0d68f69640 1 -- 172.21.15.16:0/1812967898 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d50003520 con 0x7f0d50003fd0 2024-04-24T23:49:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.527+0000 7f0d69f6b640 1 --2- 172.21.15.16:0/1812967898 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d50004d40 0x7f0d500051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.527+0000 7f0d68f69640 1 --2- 172.21.15.16:0/1812967898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50003fd0 0x7f0d50004800 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0d540098a0 tx=0x7f0d50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136ccf2acb773902 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.527+0000 7f0d5bfff640 1 -- 172.21.15.16:0/1812967898 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5400f040 con 0x7f0d50003fd0 2024-04-24T23:49:59.530 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.527+0000 7f0d5bfff640 1 -- 172.21.15.16:0/1812967898 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d5402ace0 con 0x7f0d50003fd0 2024-04-24T23:49:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.527+0000 7f0d5bfff640 1 -- 172.21.15.16:0/1812967898 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d54004250 con 0x7f0d50003fd0 2024-04-24T23:49:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.528+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/1812967898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50003fd0 msgr2=0x7f0d50004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.528+0000 7f0d6b9ce640 1 --2- 172.21.15.16:0/1812967898 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d50003fd0 0x7f0d50004800 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0d540098a0 tx=0x7f0d50005830 comp rx=0 tx=0).stop 2024-04-24T23:49:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.528+0000 7f0d69f6b640 1 -- 172.21.15.16:0/1812967898 reap_dead start 2024-04-24T23:49:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.528+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/1812967898 shutdown_connections 2024-04-24T23:49:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.528+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/1812967898 >> 172.21.15.16:0/1812967898 conn(0x7f0d640d3dc0 msgr2=0x7f0d640d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.528+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/1812967898 shutdown_connections 2024-04-24T23:49:59.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.528+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/1812967898 wait complete. 2024-04-24T23:49:59.533 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.529+0000 7f0d6b9ce640 1 Processor -- start 2024-04-24T23:49:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.529+0000 7f0d6b9ce640 1 -- start start 2024-04-24T23:49:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d6b9ce640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d50003fd0 0x7f0d6414d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d6b9ce640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d641e6b60 0x7f0d641e6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d6b9ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64160320 0x7f0d641607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d6b9ce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0d640d3050 con 0x7f0d64160320 2024-04-24T23:49:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d6b9ce640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6406ba00 con 0x7f0d641e6b60 2024-04-24T23:49:59.534 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d6b9ce640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0d640cead0 con 0x7f0d50003fd0 2024-04-24T23:49:59.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d69f6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64160320 0x7f0d641607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d69f6b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64160320 0x7f0d641607a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56700/0 (socket says 172.21.15.16:56700) 2024-04-24T23:49:59.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d69f6b640 1 -- 172.21.15.16:0/351994485 learned_addr learned my addr 172.21.15.16:0/351994485 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:59.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d68f69640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d641e6b60 0x7f0d641e6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d6976a640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d50003fd0 0x7f0d6414d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d69f6b640 1 -- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d50003fd0 msgr2=0x7f0d6414d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d69f6b640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d50003fd0 0x7f0d6414d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d69f6b640 1 -- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d641e6b60 msgr2=0x7f0d641e6fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d69f6b640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d641e6b60 0x7f0d641e6fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d69f6b640 1 -- 172.21.15.16:0/351994485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d64161280 con 0x7f0d64160320 2024-04-24T23:49:59.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.530+0000 7f0d6976a640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d50003fd0 0x7f0d6414d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:59.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.531+0000 7f0d69f6b640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64160320 0x7f0d641607a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0d64067a70 tx=0x7f0d5c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:59.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.531+0000 7f0d59ffb640 1 -- 172.21.15.16:0/351994485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c00ec10 con 0x7f0d64160320 2024-04-24T23:49:59.535 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.531+0000 7f0d59ffb640 1 -- 172.21.15.16:0/351994485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d5c018070 con 0x7f0d64160320 2024-04-24T23:49:59.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.531+0000 7f0d59ffb640 1 -- 172.21.15.16:0/351994485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d5c013420 con 0x7f0d64160320 2024-04-24T23:49:59.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.531+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/351994485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d64160ce0 con 0x7f0d64160320 2024-04-24T23:49:59.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.531+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/351994485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d640cbd70 con 0x7f0d64160320 2024-04-24T23:49:59.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.532+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/351994485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d64068f90 con 0x7f0d64160320 2024-04-24T23:49:59.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.532+0000 7f0d59ffb640 1 -- 172.21.15.16:0/351994485 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0d5c0135c0 con 0x7f0d64160320 2024-04-24T23:49:59.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.533+0000 7f0d59ffb640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d4806be60 0x7f0d4806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.533+0000 7f0d59ffb640 1 -- 172.21.15.16:0/351994485 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f0d5c08b490 con 0x7f0d64160320 2024-04-24T23:49:59.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.533+0000 7f0d6976a640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d4806be60 0x7f0d4806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.533+0000 7f0d6976a640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d4806be60 0x7f0d4806e320 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0d600036b0 tx=0x7f0d60009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:59.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.537+0000 7f0d59ffb640 1 -- 172.21.15.16:0/351994485 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0d5c059a40 con 0x7f0d64160320 2024-04-24T23:49:59.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.720+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/351994485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f0d6406b640 con 0x7f0d64160320 2024-04-24T23:49:59.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.720+0000 7f0d59ffb640 1 -- 172.21.15.16:0/351994485 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f0d5c059340 con 0x7f0d64160320 2024-04-24T23:49:59.720 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:49:59.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.722+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d4806be60 msgr2=0x7f0d4806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.722+0000 7f0d6b9ce640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d4806be60 0x7f0d4806e320 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0d600036b0 tx=0x7f0d60009040 comp rx=0 tx=0).stop 2024-04-24T23:49:59.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.722+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/351994485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64160320 msgr2=0x7f0d641607a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.722+0000 7f0d6b9ce640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64160320 0x7f0d641607a0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f0d64067a70 tx=0x7f0d5c00e530 comp rx=0 tx=0).stop 2024-04-24T23:49:59.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.722+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/351994485 shutdown_connections 2024-04-24T23:49:59.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.722+0000 7f0d6b9ce640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d4806be60 0x7f0d4806e320 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.722+0000 7f0d6b9ce640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d64160320 0x7f0d641607a0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.722+0000 7f0d6b9ce640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d641e6b60 0x7f0d641e6fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.722+0000 7f0d6b9ce640 1 --2- 172.21.15.16:0/351994485 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d50003fd0 0x7f0d6414d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.722+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/351994485 >> 172.21.15.16:0/351994485 conn(0x7f0d640d3dc0 msgr2=0x7f0d6415ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:59.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.722+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/351994485 shutdown_connections 2024-04-24T23:49:59.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.722+0000 7f0d6b9ce640 1 -- 172.21.15.16:0/351994485 wait complete. 2024-04-24T23:49:59.733 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:49:59.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.834+0000 7f02daf9e640 1 Processor -- start 2024-04-24T23:49:59.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.835+0000 7f02daf9e640 1 -- start start 2024-04-24T23:49:59.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.835+0000 7f02daf9e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02d4161890 0x7f02d406c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.835+0000 7f02daf9e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f02d40b2db0 0x7f02d40d1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.835+0000 7f02daf9e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02d40cffa0 con 0x7f02d40b2860 2024-04-24T23:49:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02daf9e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f02d406c1c0 con 0x7f02d40b2db0 2024-04-24T23:49:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02daf9e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f02d40d0210 con 0x7f02d4161890 2024-04-24T23:49:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d979b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f02d40b2860 0x7f02d4161c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58932/0 (socket says 172.21.15.16:58932) 2024-04-24T23:49:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d979b640 1 -- 172.21.15.16:0/602528063 learned_addr learned my addr 172.21.15.16:0/602528063 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:49:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02da79d640 1 --2- 172.21.15.16:0/602528063 >> v2:172.21.15.156:3300/0 conn(0x7f02d40b2db0 0x7f02d40d1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 -- 172.21.15.16:0/602528063 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 557037789 0 0) 0x7f02d40cffa0 con 0x7f02d40b2860 2024-04-24T23:49:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d9f9c640 1 --2- 172.21.15.16:0/602528063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02d4161890 0x7f02d406c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 -- 172.21.15.16:0/602528063 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f02bc003520 con 0x7f02d40b2860 2024-04-24T23:49:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 -- 172.21.15.16:0/602528063 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2426880395 0 0) 0x7f02bc003520 con 0x7f02d40b2860 2024-04-24T23:49:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 -- 172.21.15.16:0/602528063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02d4161890 msgr2=0x7f02d406c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:59.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 --2- 172.21.15.16:0/602528063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02d4161890 0x7f02d406c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 -- 172.21.15.16:0/602528063 >> v2:172.21.15.156:3300/0 conn(0x7f02d40b2db0 msgr2=0x7f02d40d1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 --2- 172.21.15.16:0/602528063 >> v2:172.21.15.156:3300/0 conn(0x7f02d40b2db0 0x7f02d40d1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 -- 172.21.15.16:0/602528063 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d406cb90 con 0x7f02d40b2860 2024-04-24T23:49:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 -- 172.21.15.16:0/602528063 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f02c8001670 con 0x7f02d40b2860 2024-04-24T23:49:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 -- 172.21.15.16:0/602528063 >> v1:172.21.15.16:6789/0 conn(0x7f02d40b2860 legacy=0x7f02d4161c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 --2- 172.21.15.16:0/602528063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02bc0039b0 0x7f02bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 --2- 172.21.15.16:0/602528063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02d4161890 0x7f02bc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 --2- 172.21.15.16:0/602528063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02bc004d40 0x7f02bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 -- 172.21.15.16:0/602528063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f02d406c1c0 con 0x7f02bc004d40 2024-04-24T23:49:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 -- 172.21.15.16:0/602528063 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f02d40d0210 con 0x7f02bc0039b0 2024-04-24T23:49:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d8f9a640 1 -- 172.21.15.16:0/602528063 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f02bc00b9c0 con 0x7f02d4161890 2024-04-24T23:49:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02da79d640 1 --2- 172.21.15.16:0/602528063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02bc004d40 0x7f02bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d9f9c640 1 --2- 172.21.15.16:0/602528063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02bc0039b0 0x7f02bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.836+0000 7f02d979b640 1 --2- 172.21.15.16:0/602528063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02d4161890 0x7f02bc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.837+0000 7f02da79d640 1 -- 172.21.15.16:0/602528063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02d4161890 msgr2=0x7f02bc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.837+0000 7f02da79d640 1 --2- 172.21.15.16:0/602528063 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02d4161890 0x7f02bc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.837+0000 7f02da79d640 1 -- 172.21.15.16:0/602528063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02bc0039b0 msgr2=0x7f02bc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:49:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.837+0000 7f02da79d640 1 --2- 172.21.15.16:0/602528063 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02bc0039b0 0x7f02bc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.837+0000 7f02da79d640 1 -- 172.21.15.16:0/602528063 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02bc003520 con 0x7f02bc004d40 2024-04-24T23:49:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.837+0000 7f02da79d640 1 --2- 172.21.15.16:0/602528063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02bc004d40 0x7f02bc0051c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009fc0 tx=0x7f02bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7bb6f06062b01c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.838+0000 7f02d8f9a640 1 -- 172.21.15.16:0/602528063 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c4009c10 con 0x7f02bc004d40 2024-04-24T23:49:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.838+0000 7f02d8f9a640 1 -- 172.21.15.16:0/602528063 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02c4036070 con 0x7f02bc004d40 2024-04-24T23:49:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.838+0000 7f02d8f9a640 1 -- 172.21.15.16:0/602528063 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c402d070 con 0x7f02bc004d40 2024-04-24T23:49:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.838+0000 7f02daf9e640 1 -- 172.21.15.16:0/602528063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02bc004d40 msgr2=0x7f02bc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.838+0000 7f02daf9e640 1 --2- 172.21.15.16:0/602528063 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02bc004d40 0x7f02bc0051c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f02c4009fc0 tx=0x7f02bc005830 comp rx=0 tx=0).stop 2024-04-24T23:49:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.838+0000 7f02da79d640 1 -- 172.21.15.16:0/602528063 reap_dead start 2024-04-24T23:49:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.838+0000 7f02daf9e640 1 -- 172.21.15.16:0/602528063 shutdown_connections 2024-04-24T23:49:59.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.838+0000 7f02daf9e640 1 -- 172.21.15.16:0/602528063 >> 172.21.15.16:0/602528063 conn(0x7f02d40d3da0 msgr2=0x7f02d40d41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:49:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.838+0000 7f02daf9e640 1 -- 172.21.15.16:0/602528063 shutdown_connections 2024-04-24T23:49:59.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.838+0000 7f02daf9e640 1 -- 172.21.15.16:0/602528063 wait complete. 2024-04-24T23:49:59.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.840+0000 7f02daf9e640 1 Processor -- start 2024-04-24T23:49:59.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.840+0000 7f02daf9e640 1 -- start start 2024-04-24T23:49:59.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.840+0000 7f02daf9e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d41f0d70 0x7f02d41f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.840+0000 7f02daf9e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d41f1150 0x7f02d415f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.840+0000 7f02daf9e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02d40cc2e0 0x7f02d415a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.840+0000 7f02daf9e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f02d415dfd0 con 0x7f02d41f0d70 2024-04-24T23:49:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.840+0000 7f02daf9e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f02d4162380 con 0x7f02d41f1150 2024-04-24T23:49:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.840+0000 7f02daf9e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f02d415e150 con 0x7f02d40cc2e0 2024-04-24T23:49:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.840+0000 7f02d9f9c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d41f0d70 0x7f02d41f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.840+0000 7f02d9f9c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d41f0d70 0x7f02d41f3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56726/0 (socket says 172.21.15.16:56726) 2024-04-24T23:49:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.840+0000 7f02d9f9c640 1 -- 172.21.15.16:0/1424755833 learned_addr learned my addr 172.21.15.16:0/1424755833 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:49:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02d979b640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d41f1150 0x7f02d415f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02da79d640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02d40cc2e0 0x7f02d415a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02d9f9c640 1 -- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02d40cc2e0 msgr2=0x7f02d415a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02d9f9c640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02d40cc2e0 0x7f02d415a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02d9f9c640 1 -- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d41f1150 msgr2=0x7f02d415f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:49:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02d9f9c640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d41f1150 0x7f02d415f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:49:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02d9f9c640 1 -- 172.21.15.16:0/1424755833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02d41f3f90 con 0x7f02d41f0d70 2024-04-24T23:49:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02d979b640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d41f1150 0x7f02d415f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:59.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02da79d640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02d40cc2e0 0x7f02d415a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:49:59.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02d9f9c640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d41f0d70 0x7f02d41f3930 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0119d0 tx=0x7f02cc011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:59.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02d27fc640 1 -- 172.21.15.16:0/1424755833 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02cc00fb50 con 0x7f02d41f0d70 2024-04-24T23:49:59.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02d27fc640 1 -- 172.21.15.16:0/1424755833 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02cc00fcf0 con 0x7f02d41f0d70 2024-04-24T23:49:59.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02daf9e640 1 -- 172.21.15.16:0/1424755833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02d41f4270 con 0x7f02d41f0d70 2024-04-24T23:49:59.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02d27fc640 1 -- 172.21.15.16:0/1424755833 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02cc00fea0 con 0x7f02d41f0d70 2024-04-24T23:49:59.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.841+0000 7f02daf9e640 1 -- 172.21.15.16:0/1424755833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02d41f4830 con 0x7f02d41f0d70 2024-04-24T23:49:59.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.842+0000 7f02daf9e640 1 -- 172.21.15.16:0/1424755833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02d404f500 con 0x7f02d41f0d70 2024-04-24T23:49:59.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.843+0000 7f02d27fc640 1 -- 172.21.15.16:0/1424755833 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f02cc012040 con 0x7f02d41f0d70 2024-04-24T23:49:59.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.843+0000 7f02d27fc640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02c006be30 0x7f02c006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:49:59.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.843+0000 7f02d27fc640 1 -- 172.21.15.16:0/1424755833 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f02cc08b8c0 con 0x7f02d41f0d70 2024-04-24T23:49:59.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.843+0000 7f02d979b640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02c006be30 0x7f02c006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:49:59.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.844+0000 7f02d979b640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02c006be30 0x7f02c006e2f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f02c8007fd0 tx=0x7f02c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:49:59.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:49:59.847+0000 7f02d27fc640 1 -- 172.21.15.16:0/1424755833 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f02cc019030 con 0x7f02d41f0d70 2024-04-24T23:50:00.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.042+0000 7f02daf9e640 1 -- 172.21.15.16:0/1424755833 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f02d40d4530 con 0x7f02d41f0d70 2024-04-24T23:50:00.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.042+0000 7f02d27fc640 1 -- 172.21.15.16:0/1424755833 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f02cc0600d0 con 0x7f02d41f0d70 2024-04-24T23:50:00.042 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-04-24T23:50:00.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.044+0000 7f02daf9e640 1 -- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02c006be30 msgr2=0x7f02c006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.044+0000 7f02daf9e640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02c006be30 0x7f02c006e2f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f02c8007fd0 tx=0x7f02c8009040 comp rx=0 tx=0).stop 2024-04-24T23:50:00.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.044+0000 7f02daf9e640 1 -- 172.21.15.16:0/1424755833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d41f0d70 msgr2=0x7f02d41f3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.044+0000 7f02daf9e640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d41f0d70 0x7f02d41f3930 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f02cc0119d0 tx=0x7f02cc011e90 comp rx=0 tx=0).stop 2024-04-24T23:50:00.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.044+0000 7f02daf9e640 1 -- 172.21.15.16:0/1424755833 shutdown_connections 2024-04-24T23:50:00.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.044+0000 7f02daf9e640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02c006be30 0x7f02c006e2f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.044+0000 7f02daf9e640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f02d40cc2e0 0x7f02d415a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.044+0000 7f02daf9e640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f02d41f1150 0x7f02d415f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.044+0000 7f02daf9e640 1 --2- 172.21.15.16:0/1424755833 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02d41f0d70 0x7f02d41f3930 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.044+0000 7f02daf9e640 1 -- 172.21.15.16:0/1424755833 >> 172.21.15.16:0/1424755833 conn(0x7f02d40d3da0 msgr2=0x7f02d40d0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:00.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.044+0000 7f02daf9e640 1 -- 172.21.15.16:0/1424755833 shutdown_connections 2024-04-24T23:50:00.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.044+0000 7f02daf9e640 1 -- 172.21.15.16:0/1424755833 wait complete. 2024-04-24T23:50:00.055 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-04-24T23:50:00.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:00.055 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-04-24T23:50:00.080 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:50:00.080 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:50:00.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:00.081 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-24T23:50:00.136 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:50:00.136 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:50:00.155 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:50:00.155 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:50:00.214 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:50:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.314+0000 7f9dcfaa5640 1 Processor -- start 2024-04-24T23:50:00.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.315+0000 7f9dcfaa5640 1 -- start start 2024-04-24T23:50:00.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.315+0000 7f9dcfaa5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9dc80b2bb0 0x7f9dc80cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.315+0000 7f9dcfaa5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dc80d1fa0 0x7f9dc80d23f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.315+0000 7f9dcfaa5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dc80b3390 con 0x7f9dc815e020 2024-04-24T23:50:00.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.315+0000 7f9dcfaa5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f9dc806bfc0 con 0x7f9dc80b2bb0 2024-04-24T23:50:00.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.316+0000 7f9dcfaa5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f9dc806c2a0 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.316+0000 7f9dcd040640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9dc80b2bb0 0x7f9dc80cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.316+0000 7f9dce042640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dc80d1fa0 0x7f9dc80d23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.316+0000 7f9dcd841640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f9dc815e020 0x7f9dc80cf270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:40742/0 (socket says 172.21.15.16:40742) 2024-04-24T23:50:00.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.316+0000 7f9dcd040640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9dc80b2bb0 0x7f9dc80cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:44862/0 (socket says 172.21.15.16:44862) 2024-04-24T23:50:00.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.316+0000 7f9dcd841640 1 -- 172.21.15.16:0/437706228 learned_addr learned my addr 172.21.15.16:0/437706228 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:00.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.316+0000 7f9dce042640 1 -- 172.21.15.16:0/437706228 >> v2:172.21.15.156:3300/0 conn(0x7f9dc80b2bb0 msgr2=0x7f9dc80cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:00.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.316+0000 7f9dce042640 1 --2- 172.21.15.16:0/437706228 >> v2:172.21.15.156:3300/0 conn(0x7f9dc80b2bb0 0x7f9dc80cf980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.316+0000 7f9dce042640 1 -- 172.21.15.16:0/437706228 >> v1:172.21.15.156:6789/0 conn(0x7f9dc815e020 legacy=0x7f9dc80cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.316+0000 7f9dce042640 1 -- 172.21.15.16:0/437706228 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc806c580 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.316+0000 7f9dcd040640 1 --2- 172.21.15.16:0/437706228 >> v2:172.21.15.156:3300/0 conn(0x7f9dc80b2bb0 0x7f9dc80cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:50:00.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.317+0000 7f9dce042640 1 --2- 172.21.15.16:0/437706228 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dc80d1fa0 0x7f9dc80d23f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0009d90 tx=0x7f9dc002a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25fff398f5a45f81 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:00.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.317+0000 7f9dcc83f640 1 -- 172.21.15.16:0/437706228 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0031600 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.317+0000 7f9dcc83f640 1 -- 172.21.15.16:0/437706228 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dc00317a0 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.317+0000 7f9dcc83f640 1 -- 172.21.15.16:0/437706228 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc0031b70 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.318+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/437706228 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dc80d1fa0 msgr2=0x7f9dc80d23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.318+0000 7f9dcfaa5640 1 --2- 172.21.15.16:0/437706228 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dc80d1fa0 0x7f9dc80d23f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9dc0009d90 tx=0x7f9dc002a5d0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.318+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/437706228 shutdown_connections 2024-04-24T23:50:00.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.318+0000 7f9dcfaa5640 1 --2- 172.21.15.16:0/437706228 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dc80d1fa0 0x7f9dc80d23f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.318+0000 7f9dcfaa5640 1 --2- 172.21.15.16:0/437706228 >> v2:172.21.15.156:3300/0 conn(0x7f9dc80b2bb0 0x7f9dc80cf980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.318+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/437706228 >> 172.21.15.16:0/437706228 conn(0x7f9dc80d3e10 msgr2=0x7f9dc80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:00.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.318+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/437706228 shutdown_connections 2024-04-24T23:50:00.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.318+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/437706228 wait complete. 2024-04-24T23:50:00.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.319+0000 7f9dcfaa5640 1 Processor -- start 2024-04-24T23:50:00.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.319+0000 7f9dcfaa5640 1 -- start start 2024-04-24T23:50:00.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcfaa5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dc80b2bb0 0x7f9dc80cc770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcfaa5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc80d1fa0 0x7f9dc80cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcfaa5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dc815e020 0x7f9dc8157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcfaa5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc815e400 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcfaa5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9dc80d1a80 con 0x7f9dc80b2bb0 2024-04-24T23:50:00.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcfaa5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f9dc80cfec0 con 0x7f9dc815e020 2024-04-24T23:50:00.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcd040640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc80d1fa0 0x7f9dc80cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcd040640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc80d1fa0 0x7f9dc80cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56742/0 (socket says 172.21.15.16:56742) 2024-04-24T23:50:00.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcd040640 1 -- 172.21.15.16:0/4082539814 learned_addr learned my addr 172.21.15.16:0/4082539814 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:00.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcd841640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dc80b2bb0 0x7f9dc80cc770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dce042640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dc815e020 0x7f9dc8157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcd040640 1 -- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dc815e020 msgr2=0x7f9dc8157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcd040640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dc815e020 0x7f9dc8157370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcd040640 1 -- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dc80b2bb0 msgr2=0x7f9dc80cc770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcd040640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dc80b2bb0 0x7f9dc80cc770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcd040640 1 -- 172.21.15.16:0/4082539814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dc80cce30 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dce042640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dc815e020 0x7f9dc8157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:00.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcd841640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dc80b2bb0 0x7f9dc80cc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:00.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.320+0000 7f9dcd040640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc80d1fa0 0x7f9dc80cb980 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9db800e960 tx=0x7f9db800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:00.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.321+0000 7f9dbe7fc640 1 -- 172.21.15.16:0/4082539814 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db8017070 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.321+0000 7f9dbe7fc640 1 -- 172.21.15.16:0/4082539814 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9db800ca00 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.321+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/4082539814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dc80cd110 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.321+0000 7f9dbe7fc640 1 -- 172.21.15.16:0/4082539814 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9db800cdd0 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.321+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/4082539814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dc80cd6d0 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.322+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/4082539814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dc80cfbd0 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.322+0000 7f9dbe7fc640 1 -- 172.21.15.16:0/4082539814 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9db801a070 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.322+0000 7f9dbe7fc640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9dac06be30 0x7f9dac06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.323+0000 7f9dbe7fc640 1 -- 172.21.15.16:0/4082539814 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f9db808b570 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.323+0000 7f9dcd841640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9dac06be30 0x7f9dac06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.323+0000 7f9dcd841640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9dac06be30 0x7f9dac06e2f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4000f60 tx=0x7f9dc4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:00.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.327+0000 7f9dbe7fc640 1 -- 172.21.15.16:0/4082539814 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9dc80cfbd0 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.509 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.509+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/4082539814 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9dc806dfe0 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.510 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.510+0000 7f9dbe7fc640 1 -- 172.21.15.16:0/4082539814 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7f9db8059a80 con 0x7f9dc80d1fa0 2024-04-24T23:50:00.510 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:00.510 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:50:00.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.512+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9dac06be30 msgr2=0x7f9dac06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.512+0000 7f9dcfaa5640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9dac06be30 0x7f9dac06e2f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4000f60 tx=0x7f9dc4009040 comp rx=0 tx=0).stop 2024-04-24T23:50:00.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.512+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/4082539814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc80d1fa0 msgr2=0x7f9dc80cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.512 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.512+0000 7f9dcfaa5640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc80d1fa0 0x7f9dc80cb980 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f9db800e960 tx=0x7f9db800ee20 comp rx=0 tx=0).stop 2024-04-24T23:50:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.512+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/4082539814 shutdown_connections 2024-04-24T23:50:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.512+0000 7f9dcfaa5640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9dac06be30 0x7f9dac06e2f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.512+0000 7f9dcfaa5640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9dc815e020 0x7f9dc8157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.512+0000 7f9dcfaa5640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9dc80d1fa0 0x7f9dc80cb980 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.512+0000 7f9dcfaa5640 1 --2- 172.21.15.16:0/4082539814 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9dc80b2bb0 0x7f9dc80cc770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.512+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/4082539814 >> 172.21.15.16:0/4082539814 conn(0x7f9dc80d3e10 msgr2=0x7f9dc80b8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.512+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/4082539814 shutdown_connections 2024-04-24T23:50:00.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.512+0000 7f9dcfaa5640 1 -- 172.21.15.16:0/4082539814 wait complete. 2024-04-24T23:50:00.523 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:50:00.524 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:50:00.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.628+0000 7f76847c4640 1 Processor -- start 2024-04-24T23:50:00.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.628+0000 7f76847c4640 1 -- start start 2024-04-24T23:50:00.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.629+0000 7f76847c4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f767c15dfd0 0x7f767c06b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.629+0000 7f76847c4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f767c06bc40 0x7f767c06c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.629+0000 7f76847c4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f767c0d24d0 con 0x7f767c0cefc0 2024-04-24T23:50:00.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.629+0000 7f76847c4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f767c0d27b0 con 0x7f767c15dfd0 2024-04-24T23:50:00.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.629+0000 7f76847c4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f767c0d2a90 con 0x7f767c06bc40 2024-04-24T23:50:00.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.630+0000 7f7682560640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f767c15dfd0 0x7f767c06b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.630+0000 7f7682d61640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f767c0cefc0 0x7f767c0cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58948/0 (socket says 172.21.15.16:58948) 2024-04-24T23:50:00.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.630+0000 7f7682d61640 1 -- 172.21.15.16:0/3773648320 learned_addr learned my addr 172.21.15.16:0/3773648320 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:00.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.630+0000 7f7681d5f640 1 --2- 172.21.15.16:0/3773648320 >> v2:172.21.15.156:3300/0 conn(0x7f767c06bc40 0x7f767c06c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.630+0000 7f7682560640 1 -- 172.21.15.16:0/3773648320 >> v2:172.21.15.156:3300/0 conn(0x7f767c06bc40 msgr2=0x7f767c06c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:00.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.630+0000 7f7682560640 1 --2- 172.21.15.16:0/3773648320 >> v2:172.21.15.156:3300/0 conn(0x7f767c06bc40 0x7f767c06c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.630+0000 7f7682560640 1 -- 172.21.15.16:0/3773648320 >> v1:172.21.15.16:6789/0 conn(0x7f767c0cefc0 legacy=0x7f767c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.630+0000 7f7682560640 1 -- 172.21.15.16:0/3773648320 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f767c0d2d70 con 0x7f767c15dfd0 2024-04-24T23:50:00.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.630+0000 7f768155e640 1 -- 172.21.15.16:0/3773648320 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 793464011 0 0) 0x7f767c0d24d0 con 0x7f767c0cefc0 2024-04-24T23:50:00.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.630+0000 7f7681d5f640 1 --2- 172.21.15.16:0/3773648320 >> v2:172.21.15.156:3300/0 conn(0x7f767c06bc40 0x7f767c06c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:00.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.631+0000 7f7682560640 1 --2- 172.21.15.16:0/3773648320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c15dfd0 0x7f767c06b2f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f767c06cf60 tx=0x7f767802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218c124e5fdbda41 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:00.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.631+0000 7f768155e640 1 -- 172.21.15.16:0/3773648320 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7678037070 con 0x7f767c15dfd0 2024-04-24T23:50:00.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.631+0000 7f768155e640 1 -- 172.21.15.16:0/3773648320 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7678032440 con 0x7f767c15dfd0 2024-04-24T23:50:00.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.631+0000 7f768155e640 1 -- 172.21.15.16:0/3773648320 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767802ae60 con 0x7f767c15dfd0 2024-04-24T23:50:00.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.631+0000 7f76847c4640 1 -- 172.21.15.16:0/3773648320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c15dfd0 msgr2=0x7f767c06b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.631+0000 7f76847c4640 1 --2- 172.21.15.16:0/3773648320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c15dfd0 0x7f767c06b2f0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f767c06cf60 tx=0x7f767802a110 comp rx=0 tx=0).stop 2024-04-24T23:50:00.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.631+0000 7f76847c4640 1 -- 172.21.15.16:0/3773648320 shutdown_connections 2024-04-24T23:50:00.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.631+0000 7f76847c4640 1 --2- 172.21.15.16:0/3773648320 >> v2:172.21.15.156:3300/0 conn(0x7f767c06bc40 0x7f767c06c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.631+0000 7f76847c4640 1 --2- 172.21.15.16:0/3773648320 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c15dfd0 0x7f767c06b2f0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.631+0000 7f76847c4640 1 -- 172.21.15.16:0/3773648320 >> 172.21.15.16:0/3773648320 conn(0x7f767c0d3dc0 msgr2=0x7f767c0d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:00.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.631+0000 7f76847c4640 1 -- 172.21.15.16:0/3773648320 shutdown_connections 2024-04-24T23:50:00.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.631+0000 7f76847c4640 1 -- 172.21.15.16:0/3773648320 wait complete. 2024-04-24T23:50:00.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.633+0000 7f76847c4640 1 Processor -- start 2024-04-24T23:50:00.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.633+0000 7f76847c4640 1 -- start start 2024-04-24T23:50:00.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.633+0000 7f76847c4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c06bc40 0x7f767c0c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.633+0000 7f76847c4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f767c0cefc0 0x7f767c159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.633+0000 7f76847c4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f767c15dfd0 0x7f767c0ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.633+0000 7f76847c4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f767c06b830 con 0x7f767c06bc40 2024-04-24T23:50:00.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.633+0000 7f76847c4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f767c0cead0 con 0x7f767c0cefc0 2024-04-24T23:50:00.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.633+0000 7f76847c4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f767c06b9b0 con 0x7f767c15dfd0 2024-04-24T23:50:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.633+0000 7f7682560640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c06bc40 0x7f767c0c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.633+0000 7f7682560640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c06bc40 0x7f767c0c9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56766/0 (socket says 172.21.15.16:56766) 2024-04-24T23:50:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.633+0000 7f7682560640 1 -- 172.21.15.16:0/2899674202 learned_addr learned my addr 172.21.15.16:0/2899674202 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f7681d5f640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f767c0cefc0 0x7f767c159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f7682d61640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f767c15dfd0 0x7f767c0ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f7682560640 1 -- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f767c15dfd0 msgr2=0x7f767c0ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f7682560640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f767c15dfd0 0x7f767c0ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f7682560640 1 -- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f767c0cefc0 msgr2=0x7f767c159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f7682560640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f767c0cefc0 0x7f767c159b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f7682560640 1 -- 172.21.15.16:0/2899674202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f767c158da0 con 0x7f767c06bc40 2024-04-24T23:50:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f7681d5f640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f767c0cefc0 0x7f767c159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:00.638 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f7682d61640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f767c15dfd0 0x7f767c0ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f7682560640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c06bc40 0x7f767c0c9bd0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7678000c00 tx=0x7f76780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f7672ffd640 1 -- 172.21.15.16:0/2899674202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7678037040 con 0x7f767c06bc40 2024-04-24T23:50:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f7672ffd640 1 -- 172.21.15.16:0/2899674202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7678042070 con 0x7f767c06bc40 2024-04-24T23:50:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f76847c4640 1 -- 172.21.15.16:0/2899674202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f767c159080 con 0x7f767c06bc40 2024-04-24T23:50:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f7672ffd640 1 -- 172.21.15.16:0/2899674202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7678030bd0 con 0x7f767c06bc40 2024-04-24T23:50:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.634+0000 7f76847c4640 1 -- 172.21.15.16:0/2899674202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f767c159640 con 0x7f767c06bc40 2024-04-24T23:50:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.635+0000 7f76847c4640 1 -- 172.21.15.16:0/2899674202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f767c068f90 con 0x7f767c06bc40 2024-04-24T23:50:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.635+0000 7f7672ffd640 1 -- 172.21.15.16:0/2899674202 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7678047460 con 0x7f767c06bc40 2024-04-24T23:50:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.636+0000 7f7672ffd640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f766006be30 0x7f766006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.636+0000 7f7672ffd640 1 -- 172.21.15.16:0/2899674202 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f76780abec0 con 0x7f767c06bc40 2024-04-24T23:50:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.636+0000 7f7681d5f640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f766006be30 0x7f766006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.637+0000 7f7681d5f640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f766006be30 0x7f766006e2f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f766c006fd0 tx=0x7f766c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:00.640 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.640+0000 7f7672ffd640 1 -- 172.21.15.16:0/2899674202 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7678032e20 con 0x7f767c06bc40 2024-04-24T23:50:00.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.823+0000 7f76847c4640 1 -- 172.21.15.16:0/2899674202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f767c0b6990 con 0x7f767c06bc40 2024-04-24T23:50:00.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.823+0000 7f7672ffd640 1 -- 172.21.15.16:0/2899674202 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f767807a3e0 con 0x7f767c06bc40 2024-04-24T23:50:00.823 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:50:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.825+0000 7f76847c4640 1 -- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f766006be30 msgr2=0x7f766006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.825+0000 7f76847c4640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f766006be30 0x7f766006e2f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f766c006fd0 tx=0x7f766c008040 comp rx=0 tx=0).stop 2024-04-24T23:50:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.825+0000 7f76847c4640 1 -- 172.21.15.16:0/2899674202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c06bc40 msgr2=0x7f767c0c9bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.825 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.825+0000 7f76847c4640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c06bc40 0x7f767c0c9bd0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f7678000c00 tx=0x7f76780046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.825+0000 7f76847c4640 1 -- 172.21.15.16:0/2899674202 shutdown_connections 2024-04-24T23:50:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.825+0000 7f76847c4640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f766006be30 0x7f766006e2f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.825+0000 7f76847c4640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f767c15dfd0 0x7f767c0ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.825+0000 7f76847c4640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f767c0cefc0 0x7f767c159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.825+0000 7f76847c4640 1 --2- 172.21.15.16:0/2899674202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f767c06bc40 0x7f767c0c9bd0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.825+0000 7f76847c4640 1 -- 172.21.15.16:0/2899674202 >> 172.21.15.16:0/2899674202 conn(0x7f767c0d3dc0 msgr2=0x7f767c0b6f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.825+0000 7f76847c4640 1 -- 172.21.15.16:0/2899674202 shutdown_connections 2024-04-24T23:50:00.826 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.825+0000 7f76847c4640 1 -- 172.21.15.16:0/2899674202 wait complete. 2024-04-24T23:50:00.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:50:00.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.940+0000 7f793916a640 1 Processor -- start 2024-04-24T23:50:00.940 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.940+0000 7f793916a640 1 -- start start 2024-04-24T23:50:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.941+0000 7f793916a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f79341478d0 0x7f79340d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.941+0000 7f793916a640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f79340d2090 0x7f79340b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.941+0000 7f793916a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79340b2a10 con 0x7f79340b21d0 2024-04-24T23:50:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.941+0000 7f793916a640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f79340d24f0 con 0x7f79340d2090 2024-04-24T23:50:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.941+0000 7f793916a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f79340b3620 con 0x7f79341478d0 2024-04-24T23:50:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.941+0000 7f7932d9d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f79341478d0 0x7f79340d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.941+0000 7f792a59c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f79340d2090 0x7f79340b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.941+0000 7f7932d9d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f79341478d0 0x7f79340d1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:56494/0 (socket says 172.21.15.16:56494) 2024-04-24T23:50:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.941+0000 7f7932d9d640 1 -- 172.21.15.16:0/647282055 learned_addr learned my addr 172.21.15.16:0/647282055 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.941+0000 7f793359e640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7f79340b21d0 0x7f79340d0340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:40756/0 (socket says 172.21.15.16:40756) 2024-04-24T23:50:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.941+0000 7f7932d9d640 1 -- 172.21.15.16:0/647282055 >> v2:172.21.15.156:3300/0 conn(0x7f79340d2090 msgr2=0x7f79340b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.941+0000 7f7932d9d640 1 --2- 172.21.15.16:0/647282055 >> v2:172.21.15.156:3300/0 conn(0x7f79340d2090 0x7f79340b1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.941+0000 7f7932d9d640 1 -- 172.21.15.16:0/647282055 >> v1:172.21.15.156:6789/0 conn(0x7f79340b21d0 legacy=0x7f79340d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.941+0000 7f7932d9d640 1 -- 172.21.15.16:0/647282055 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79340b2cf0 con 0x7f79341478d0 2024-04-24T23:50:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.942+0000 7f7932d9d640 1 --2- 172.21.15.16:0/647282055 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f79341478d0 0x7f79340d1b50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7924009870 tx=0x7f792402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88878b7ed39265a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.942+0000 7f793259c640 1 -- 172.21.15.16:0/647282055 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7924037070 con 0x7f79341478d0 2024-04-24T23:50:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.942+0000 7f793259c640 1 -- 172.21.15.16:0/647282055 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7924032440 con 0x7f79341478d0 2024-04-24T23:50:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.942+0000 7f793259c640 1 -- 172.21.15.16:0/647282055 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7924032810 con 0x7f79341478d0 2024-04-24T23:50:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.943+0000 7f793916a640 1 -- 172.21.15.16:0/647282055 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f79341478d0 msgr2=0x7f79340d1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.943+0000 7f793916a640 1 --2- 172.21.15.16:0/647282055 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f79341478d0 0x7f79340d1b50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7924009870 tx=0x7f792402a270 comp rx=0 tx=0).stop 2024-04-24T23:50:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.943+0000 7f793916a640 1 -- 172.21.15.16:0/647282055 shutdown_connections 2024-04-24T23:50:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.943+0000 7f793916a640 1 --2- 172.21.15.16:0/647282055 >> v2:172.21.15.156:3300/0 conn(0x7f79340d2090 0x7f79340b1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.943+0000 7f793916a640 1 --2- 172.21.15.16:0/647282055 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f79341478d0 0x7f79340d1b50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.943+0000 7f793916a640 1 -- 172.21.15.16:0/647282055 >> 172.21.15.16:0/647282055 conn(0x7f793406b2c0 msgr2=0x7f7934072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:00.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.943+0000 7f793916a640 1 -- 172.21.15.16:0/647282055 shutdown_connections 2024-04-24T23:50:00.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.943+0000 7f793916a640 1 -- 172.21.15.16:0/647282055 wait complete. 2024-04-24T23:50:00.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.944+0000 7f793916a640 1 Processor -- start 2024-04-24T23:50:00.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.944+0000 7f793916a640 1 -- start start 2024-04-24T23:50:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.945+0000 7f793916a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79340b21d0 0x7f793415d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.945+0000 7f793916a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79340d2090 0x7f793413b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.945+0000 7f793916a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f79341478d0 0x7f7934160900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.945+0000 7f793916a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f79340b2660 con 0x7f79340b21d0 2024-04-24T23:50:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.945+0000 7f793916a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f79340b32b0 con 0x7f79340d2090 2024-04-24T23:50:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.945+0000 7f793916a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f79341402e0 con 0x7f79341478d0 2024-04-24T23:50:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.945+0000 7f7932d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79340b21d0 0x7f793415d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.945+0000 7f7932d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79340b21d0 0x7f793415d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56768/0 (socket says 172.21.15.16:56768) 2024-04-24T23:50:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.945+0000 7f7932d9d640 1 -- 172.21.15.16:0/446950576 learned_addr learned my addr 172.21.15.16:0/446950576 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.946+0000 7f793359e640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f79341478d0 0x7f7934160900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.946+0000 7f792a59c640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79340d2090 0x7f793413b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.946+0000 7f7932d9d640 1 -- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f79341478d0 msgr2=0x7f7934160900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.946+0000 7f7932d9d640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f79341478d0 0x7f7934160900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.946+0000 7f7932d9d640 1 -- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79340d2090 msgr2=0x7f793413b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.946+0000 7f7932d9d640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79340d2090 0x7f793413b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:00.951 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.946+0000 7f7932d9d640 1 -- 172.21.15.16:0/446950576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7934140580 con 0x7f79340b21d0 2024-04-24T23:50:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.946+0000 7f792a59c640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79340d2090 0x7f793413b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.946+0000 7f7932d9d640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79340b21d0 0x7f793415d810 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7924000c00 tx=0x7f79240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.946+0000 7f792bfff640 1 -- 172.21.15.16:0/446950576 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7924041070 con 0x7f79340b21d0 2024-04-24T23:50:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.946+0000 7f792bfff640 1 -- 172.21.15.16:0/446950576 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f792403c890 con 0x7f79340b21d0 2024-04-24T23:50:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.946+0000 7f793916a640 1 -- 172.21.15.16:0/446950576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7934140860 con 0x7f79340b21d0 2024-04-24T23:50:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.946+0000 7f792bfff640 1 -- 172.21.15.16:0/446950576 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7924037040 con 0x7f79340b21d0 2024-04-24T23:50:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.946+0000 7f793916a640 1 -- 172.21.15.16:0/446950576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f793413bc50 con 0x7f79340b21d0 2024-04-24T23:50:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.947+0000 7f793916a640 1 -- 172.21.15.16:0/446950576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79340d1510 con 0x7f79340b21d0 2024-04-24T23:50:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.948+0000 7f792bfff640 1 -- 172.21.15.16:0/446950576 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7924044070 con 0x7f79340b21d0 2024-04-24T23:50:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.948+0000 7f792bfff640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f791406be30 0x7f791406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.948+0000 7f792bfff640 1 -- 172.21.15.16:0/446950576 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f79240abd30 con 0x7f79340b21d0 2024-04-24T23:50:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.948+0000 7f792a59c640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f791406be30 0x7f791406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:00.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.949+0000 7f792a59c640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f791406be30 0x7f791406e2f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f79340b1670 tx=0x7f791c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:00.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:00.953+0000 7f792bfff640 1 -- 172.21.15.16:0/446950576 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f79340d1510 con 0x7f79340b21d0 2024-04-24T23:50:01.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.137+0000 7f793916a640 1 -- 172.21.15.16:0/446950576 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7934068f90 con 0x7f79340b21d0 2024-04-24T23:50:01.137 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.137+0000 7f792bfff640 1 -- 172.21.15.16:0/446950576 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7f7924079e40 con 0x7f79340b21d0 2024-04-24T23:50:01.137 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:50:01.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.139+0000 7f793916a640 1 -- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f791406be30 msgr2=0x7f791406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.139+0000 7f793916a640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f791406be30 0x7f791406e2f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f79340b1670 tx=0x7f791c008040 comp rx=0 tx=0).stop 2024-04-24T23:50:01.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.139+0000 7f793916a640 1 -- 172.21.15.16:0/446950576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79340b21d0 msgr2=0x7f793415d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.139+0000 7f793916a640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79340b21d0 0x7f793415d810 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f7924000c00 tx=0x7f79240046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.139+0000 7f793916a640 1 -- 172.21.15.16:0/446950576 shutdown_connections 2024-04-24T23:50:01.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.139+0000 7f793916a640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f791406be30 0x7f791406e2f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.139+0000 7f793916a640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f79341478d0 0x7f7934160900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.139+0000 7f793916a640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f79340d2090 0x7f793413b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.139+0000 7f793916a640 1 --2- 172.21.15.16:0/446950576 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79340b21d0 0x7f793415d810 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.139+0000 7f793916a640 1 -- 172.21.15.16:0/446950576 >> 172.21.15.16:0/446950576 conn(0x7f793406b2c0 msgr2=0x7f79340d0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:01.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.139+0000 7f793916a640 1 -- 172.21.15.16:0/446950576 shutdown_connections 2024-04-24T23:50:01.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.139+0000 7f793916a640 1 -- 172.21.15.16:0/446950576 wait complete. 2024-04-24T23:50:01.150 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:50:01.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.256+0000 7f16a219d640 1 Processor -- start 2024-04-24T23:50:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.256+0000 7f16a219d640 1 -- start start 2024-04-24T23:50:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f16a219d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f169c0d2260 0x7f169c0b1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f16a219d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169c0b2a10 con 0x7f169c0b21d0 2024-04-24T23:50:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f16a219d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169c0b2cf0 con 0x7f169c1478d0 2024-04-24T23:50:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f16a219d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f169c0b3620 con 0x7f169c0d2260 2024-04-24T23:50:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f169c0b21d0 0x7f169c0b4e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58964/0 (socket says 172.21.15.16:58964) 2024-04-24T23:50:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169bfff640 1 -- 172.21.15.16:0/2881225556 learned_addr learned my addr 172.21.15.16:0/2881225556 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169affd640 1 --2- 172.21.15.16:0/2881225556 >> v2:172.21.15.156:3300/0 conn(0x7f169c0d2260 0x7f169c0b1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169a7fc640 1 -- 172.21.15.16:0/2881225556 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 978287670 0 0) 0x7f169c0b2a10 con 0x7f169c0b21d0 2024-04-24T23:50:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169a7fc640 1 -- 172.21.15.16:0/2881225556 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1684003520 con 0x7f169c0b21d0 2024-04-24T23:50:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169a7fc640 1 -- 172.21.15.16:0/2881225556 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3708584846 0 0) 0x7f1684003520 con 0x7f169c0b21d0 2024-04-24T23:50:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169a7fc640 1 -- 172.21.15.16:0/2881225556 >> v2:172.21.15.156:3300/0 conn(0x7f169c0d2260 msgr2=0x7f169c0b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169a7fc640 1 --2- 172.21.15.16:0/2881225556 >> v2:172.21.15.156:3300/0 conn(0x7f169c0d2260 0x7f169c0b1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169a7fc640 1 -- 172.21.15.16:0/2881225556 >> v1:172.21.15.156:6789/0 conn(0x7f169c1478d0 legacy=0x7f169c0d1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169a7fc640 1 -- 172.21.15.16:0/2881225556 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c0b2fd0 con 0x7f169c0b21d0 2024-04-24T23:50:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169a7fc640 1 -- 172.21.15.16:0/2881225556 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f16940019f0 con 0x7f169c0b21d0 2024-04-24T23:50:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169a7fc640 1 -- 172.21.15.16:0/2881225556 >> v1:172.21.15.16:6789/0 conn(0x7f169c0b21d0 legacy=0x7f169c0b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169a7fc640 1 --2- 172.21.15.16:0/2881225556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16840039b0 0x7f1684003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169a7fc640 1 --2- 172.21.15.16:0/2881225556 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f16840043b0 0x7f1684004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.257+0000 7f169a7fc640 1 --2- 172.21.15.16:0/2881225556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1684005080 0x7f168400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.258+0000 7f169a7fc640 1 -- 172.21.15.16:0/2881225556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f169c0b3620 con 0x7f16840039b0 2024-04-24T23:50:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.258+0000 7f169a7fc640 1 -- 172.21.15.16:0/2881225556 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f168400bfe0 con 0x7f1684005080 2024-04-24T23:50:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.258+0000 7f169a7fc640 1 -- 172.21.15.16:0/2881225556 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f168400c1e0 con 0x7f16840043b0 2024-04-24T23:50:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.258+0000 7f169b7fe640 1 --2- 172.21.15.16:0/2881225556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16840039b0 0x7f1684003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.258+0000 7f169bfff640 1 --2- 172.21.15.16:0/2881225556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1684005080 0x7f168400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.258+0000 7f169affd640 1 --2- 172.21.15.16:0/2881225556 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f16840043b0 0x7f1684004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.258+0000 7f169b7fe640 1 -- 172.21.15.16:0/2881225556 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f16840043b0 msgr2=0x7f1684004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.258+0000 7f169b7fe640 1 --2- 172.21.15.16:0/2881225556 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f16840043b0 0x7f1684004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.258+0000 7f169b7fe640 1 -- 172.21.15.16:0/2881225556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1684005080 msgr2=0x7f168400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.258+0000 7f169b7fe640 1 --2- 172.21.15.16:0/2881225556 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1684005080 0x7f168400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.258+0000 7f169b7fe640 1 -- 172.21.15.16:0/2881225556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1684003520 con 0x7f16840039b0 2024-04-24T23:50:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.258+0000 7f169bfff640 1 -- 172.21.15.16:0/2881225556 reap_dead start 2024-04-24T23:50:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.259+0000 7f169b7fe640 1 --2- 172.21.15.16:0/2881225556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16840039b0 0x7f1684003d90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1690001860 tx=0x7f1684005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd9e0486ac142dc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.259+0000 7f169a7fc640 1 -- 172.21.15.16:0/2881225556 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1690037070 con 0x7f16840039b0 2024-04-24T23:50:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.259+0000 7f169a7fc640 1 -- 172.21.15.16:0/2881225556 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f169002abf0 con 0x7f16840039b0 2024-04-24T23:50:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.259+0000 7f169a7fc640 1 -- 172.21.15.16:0/2881225556 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1690001430 con 0x7f16840039b0 2024-04-24T23:50:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.259+0000 7f16a219d640 1 -- 172.21.15.16:0/2881225556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16840039b0 msgr2=0x7f1684003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.259+0000 7f16a219d640 1 --2- 172.21.15.16:0/2881225556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16840039b0 0x7f1684003d90 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f1690001860 tx=0x7f1684005830 comp rx=0 tx=0).stop 2024-04-24T23:50:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.259+0000 7f16a219d640 1 -- 172.21.15.16:0/2881225556 shutdown_connections 2024-04-24T23:50:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.259+0000 7f16a219d640 1 --2- 172.21.15.16:0/2881225556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16840039b0 0x7f1684003d90 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.259+0000 7f16a219d640 1 -- 172.21.15.16:0/2881225556 >> 172.21.15.16:0/2881225556 conn(0x7f169c06b2c0 msgr2=0x7f169c072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.259+0000 7f16a219d640 1 -- 172.21.15.16:0/2881225556 shutdown_connections 2024-04-24T23:50:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.259+0000 7f16a219d640 1 -- 172.21.15.16:0/2881225556 wait complete. 2024-04-24T23:50:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.261+0000 7f16a219d640 1 Processor -- start 2024-04-24T23:50:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.261+0000 7f16a219d640 1 -- start start 2024-04-24T23:50:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.261+0000 7f16a219d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f16840039b0 0x7f169c160f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.261+0000 7f16a219d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f169c13ba80 0x7f169c13ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.261+0000 7f16a219d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f169c13cb70 0x7f169c13a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.261+0000 7f16a219d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f169c0b27a0 con 0x7f169c13ba80 2024-04-24T23:50:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.261+0000 7f16a219d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f169c0b2620 con 0x7f169c13cb70 2024-04-24T23:50:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.261+0000 7f16a219d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f169c0b1c90 con 0x7f16840039b0 2024-04-24T23:50:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f169affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f169c13ba80 0x7f169c13ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f169affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f169c13ba80 0x7f169c13ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56786/0 (socket says 172.21.15.16:56786) 2024-04-24T23:50:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f169affd640 1 -- 172.21.15.16:0/2534196256 learned_addr learned my addr 172.21.15.16:0/2534196256 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:01.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f169b7fe640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f16840039b0 0x7f169c160f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f169bfff640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f169c13cb70 0x7f169c13a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f169affd640 1 -- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f16840039b0 msgr2=0x7f169c160f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f169affd640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f16840039b0 0x7f169c160f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f169affd640 1 -- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f169c13cb70 msgr2=0x7f169c13a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f169affd640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f169c13cb70 0x7f169c13a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f169affd640 1 -- 172.21.15.16:0/2534196256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c15e500 con 0x7f169c13ba80 2024-04-24T23:50:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f169b7fe640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f16840039b0 0x7f169c160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f169bfff640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f169c13cb70 0x7f169c13a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f169affd640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f169c13ba80 0x7f169c13ecd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f168c011990 tx=0x7f168c011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f1683fff640 1 -- 172.21.15.16:0/2534196256 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c00fad0 con 0x7f169c13ba80 2024-04-24T23:50:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f1683fff640 1 -- 172.21.15.16:0/2534196256 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f168c00fc70 con 0x7f169c13ba80 2024-04-24T23:50:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f1683fff640 1 -- 172.21.15.16:0/2534196256 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c014440 con 0x7f169c13ba80 2024-04-24T23:50:01.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f16a219d640 1 -- 172.21.15.16:0/2534196256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f169c15e7e0 con 0x7f169c13ba80 2024-04-24T23:50:01.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.262+0000 7f16a219d640 1 -- 172.21.15.16:0/2534196256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f169c13bee0 con 0x7f169c13ba80 2024-04-24T23:50:01.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.263+0000 7f16a219d640 1 -- 172.21.15.16:0/2534196256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f169c068f90 con 0x7f169c13ba80 2024-04-24T23:50:01.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.264+0000 7f1683fff640 1 -- 172.21.15.16:0/2534196256 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f168c012040 con 0x7f169c13ba80 2024-04-24T23:50:01.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.264+0000 7f1683fff640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f168806be30 0x7f168806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.264+0000 7f1683fff640 1 -- 172.21.15.16:0/2534196256 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f168c08b8a0 con 0x7f169c13ba80 2024-04-24T23:50:01.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.264+0000 7f169b7fe640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f168806be30 0x7f168806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.265+0000 7f169b7fe640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f168806be30 0x7f168806e2f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1690034000 tx=0x7f1690009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:01.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.268+0000 7f1683fff640 1 -- 172.21.15.16:0/2534196256 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f168c019030 con 0x7f169c13ba80 2024-04-24T23:50:01.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.448+0000 7f16a219d640 1 -- 172.21.15.16:0/2534196256 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f169c070800 con 0x7f169c13ba80 2024-04-24T23:50:01.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.449+0000 7f1683fff640 1 -- 172.21.15.16:0/2534196256 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f168c059e50 con 0x7f169c13ba80 2024-04-24T23:50:01.449 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:50:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.451+0000 7f16a219d640 1 -- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f168806be30 msgr2=0x7f168806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.451+0000 7f16a219d640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f168806be30 0x7f168806e2f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1690034000 tx=0x7f1690009030 comp rx=0 tx=0).stop 2024-04-24T23:50:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.451+0000 7f16a219d640 1 -- 172.21.15.16:0/2534196256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f169c13ba80 msgr2=0x7f169c13ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.451+0000 7f16a219d640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f169c13ba80 0x7f169c13ecd0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f168c011990 tx=0x7f168c011e50 comp rx=0 tx=0).stop 2024-04-24T23:50:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.451+0000 7f16a219d640 1 -- 172.21.15.16:0/2534196256 shutdown_connections 2024-04-24T23:50:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.451+0000 7f16a219d640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f168806be30 0x7f168806e2f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.451+0000 7f16a219d640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f169c13cb70 0x7f169c13a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.451+0000 7f16a219d640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f169c13ba80 0x7f169c13ecd0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.451+0000 7f16a219d640 1 --2- 172.21.15.16:0/2534196256 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f16840039b0 0x7f169c160f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.451+0000 7f16a219d640 1 -- 172.21.15.16:0/2534196256 >> 172.21.15.16:0/2534196256 conn(0x7f169c06b2c0 msgr2=0x7f169c0700f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.451+0000 7f16a219d640 1 -- 172.21.15.16:0/2534196256 shutdown_connections 2024-04-24T23:50:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.451+0000 7f16a219d640 1 -- 172.21.15.16:0/2534196256 wait complete. 2024-04-24T23:50:01.461 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:50:01.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.566+0000 7f6906fb0640 1 Processor -- start 2024-04-24T23:50:01.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.566+0000 7f6906fb0640 1 -- start start 2024-04-24T23:50:01.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.566+0000 7f6906fb0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f690015dfd0 0x7f690006b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.566+0000 7f6906fb0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f690006bc40 0x7f690006c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.566+0000 7f6906fb0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69000cefc0 0x7f69000cf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.566+0000 7f6906fb0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f690006c5e0 con 0x7f69000cefc0 2024-04-24T23:50:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.566+0000 7f6906fb0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f69000d24d0 con 0x7f690015dfd0 2024-04-24T23:50:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.566+0000 7f6906fb0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f69000d27b0 con 0x7f690006bc40 2024-04-24T23:50:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.567+0000 7f690554d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69000cefc0 0x7f69000cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.567+0000 7f690554d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69000cefc0 0x7f69000cf410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56798/0 (socket says 172.21.15.16:56798) 2024-04-24T23:50:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.567+0000 7f690554d640 1 -- 172.21.15.16:0/2796752656 learned_addr learned my addr 172.21.15.16:0/2796752656 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.567+0000 7f68fffff640 1 --2- 172.21.15.16:0/2796752656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f690006bc40 0x7f690006c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.567+0000 7f6904d4c640 1 --2- 172.21.15.16:0/2796752656 >> v2:172.21.15.156:3300/0 conn(0x7f690015dfd0 0x7f690006b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.567+0000 7f690554d640 1 -- 172.21.15.16:0/2796752656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f690006bc40 msgr2=0x7f690006c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.567+0000 7f690554d640 1 --2- 172.21.15.16:0/2796752656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f690006bc40 0x7f690006c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.567+0000 7f690554d640 1 -- 172.21.15.16:0/2796752656 >> v2:172.21.15.156:3300/0 conn(0x7f690015dfd0 msgr2=0x7f690006b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.567+0000 7f690554d640 1 --2- 172.21.15.16:0/2796752656 >> v2:172.21.15.156:3300/0 conn(0x7f690015dfd0 0x7f690006b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.567+0000 7f690554d640 1 -- 172.21.15.16:0/2796752656 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69000d2a90 con 0x7f69000cefc0 2024-04-24T23:50:01.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.567+0000 7f68fffff640 1 --2- 172.21.15.16:0/2796752656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f690006bc40 0x7f690006c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:01.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.568+0000 7f690554d640 1 --2- 172.21.15.16:0/2796752656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69000cefc0 0x7f69000cf410 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f69000d3ce0 tx=0x7f68f802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1116f2feaefacc3f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:01.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.568+0000 7f68ff7fe640 1 -- 172.21.15.16:0/2796752656 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f8036070 con 0x7f69000cefc0 2024-04-24T23:50:01.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.568+0000 7f68ff7fe640 1 -- 172.21.15.16:0/2796752656 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f8031600 con 0x7f69000cefc0 2024-04-24T23:50:01.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.568+0000 7f68ff7fe640 1 -- 172.21.15.16:0/2796752656 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f80319d0 con 0x7f69000cefc0 2024-04-24T23:50:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.568+0000 7f6906fb0640 1 -- 172.21.15.16:0/2796752656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69000cefc0 msgr2=0x7f69000cf410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.568+0000 7f6906fb0640 1 --2- 172.21.15.16:0/2796752656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69000cefc0 0x7f69000cf410 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f69000d3ce0 tx=0x7f68f802a610 comp rx=0 tx=0).stop 2024-04-24T23:50:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.568+0000 7f6906fb0640 1 -- 172.21.15.16:0/2796752656 shutdown_connections 2024-04-24T23:50:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.568+0000 7f6906fb0640 1 --2- 172.21.15.16:0/2796752656 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69000cefc0 0x7f69000cf410 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.568+0000 7f6906fb0640 1 --2- 172.21.15.16:0/2796752656 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f690006bc40 0x7f690006c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.568+0000 7f6906fb0640 1 --2- 172.21.15.16:0/2796752656 >> v2:172.21.15.156:3300/0 conn(0x7f690015dfd0 0x7f690006b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.568+0000 7f6906fb0640 1 -- 172.21.15.16:0/2796752656 >> 172.21.15.16:0/2796752656 conn(0x7f69000d3dc0 msgr2=0x7f69000d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.568+0000 7f6906fb0640 1 -- 172.21.15.16:0/2796752656 shutdown_connections 2024-04-24T23:50:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.568+0000 7f6906fb0640 1 -- 172.21.15.16:0/2796752656 wait complete. 2024-04-24T23:50:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.570+0000 7f6906fb0640 1 Processor -- start 2024-04-24T23:50:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.570+0000 7f6906fb0640 1 -- start start 2024-04-24T23:50:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.570+0000 7f6906fb0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f690006bc40 0x7f690014d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.570+0000 7f6906fb0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69000cefc0 0x7f69000cbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.570+0000 7f6906fb0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690015dfd0 0x7f690015ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.570+0000 7f6906fb0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f69000cead0 con 0x7f690015dfd0 2024-04-24T23:50:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.570+0000 7f6906fb0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f69000d3050 con 0x7f69000cefc0 2024-04-24T23:50:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.570+0000 7f6906fb0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f690006b830 con 0x7f690006bc40 2024-04-24T23:50:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f690554d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690015dfd0 0x7f690015ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f690554d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690015dfd0 0x7f690015ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56806/0 (socket says 172.21.15.16:56806) 2024-04-24T23:50:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f690554d640 1 -- 172.21.15.16:0/960908050 learned_addr learned my addr 172.21.15.16:0/960908050 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f6904d4c640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f690006bc40 0x7f690014d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f68fffff640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69000cefc0 0x7f69000cbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f690554d640 1 -- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f690006bc40 msgr2=0x7f690014d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f690554d640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f690006bc40 0x7f690014d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f690554d640 1 -- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69000cefc0 msgr2=0x7f69000cbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f690554d640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69000cefc0 0x7f69000cbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f690554d640 1 -- 172.21.15.16:0/960908050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69000cc800 con 0x7f690015dfd0 2024-04-24T23:50:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f68fffff640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69000cefc0 0x7f69000cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f690554d640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690015dfd0 0x7f690015ae60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f68f8009f40 tx=0x7f68f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f68fd7fa640 1 -- 172.21.15.16:0/960908050 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f8036050 con 0x7f690015dfd0 2024-04-24T23:50:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f68fd7fa640 1 -- 172.21.15.16:0/960908050 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68f8042070 con 0x7f690015dfd0 2024-04-24T23:50:01.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f6906fb0640 1 -- 172.21.15.16:0/960908050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69000cc520 con 0x7f690015dfd0 2024-04-24T23:50:01.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f68fd7fa640 1 -- 172.21.15.16:0/960908050 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68f8009510 con 0x7f690015dfd0 2024-04-24T23:50:01.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.571+0000 7f6906fb0640 1 -- 172.21.15.16:0/960908050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6900158f60 con 0x7f690015dfd0 2024-04-24T23:50:01.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.572+0000 7f6906fb0640 1 -- 172.21.15.16:0/960908050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6900068f90 con 0x7f690015dfd0 2024-04-24T23:50:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.573+0000 7f68fd7fa640 1 -- 172.21.15.16:0/960908050 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f68f8007bd0 con 0x7f690015dfd0 2024-04-24T23:50:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.573+0000 7f68fd7fa640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f68e406be30 0x7f68e406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.573+0000 7f68fd7fa640 1 -- 172.21.15.16:0/960908050 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f68f80ac1f0 con 0x7f690015dfd0 2024-04-24T23:50:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.574+0000 7f6904d4c640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f68e406be30 0x7f68e406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.574+0000 7f6904d4c640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f68e406be30 0x7f68e406e2f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f690015aac0 tx=0x7f68f4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.578+0000 7f68fd7fa640 1 -- 172.21.15.16:0/960908050 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f68f807a7a0 con 0x7f690015dfd0 2024-04-24T23:50:01.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.760+0000 7f6906fb0640 1 -- 172.21.15.16:0/960908050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f69000ce1f0 con 0x7f690015dfd0 2024-04-24T23:50:01.760 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.760+0000 7f68fd7fa640 1 -- 172.21.15.16:0/960908050 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f68f807a0a0 con 0x7f690015dfd0 2024-04-24T23:50:01.760 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:50:01.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.762+0000 7f6906fb0640 1 -- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f68e406be30 msgr2=0x7f68e406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.762+0000 7f6906fb0640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f68e406be30 0x7f68e406e2f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f690015aac0 tx=0x7f68f4008040 comp rx=0 tx=0).stop 2024-04-24T23:50:01.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.762+0000 7f6906fb0640 1 -- 172.21.15.16:0/960908050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690015dfd0 msgr2=0x7f690015ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.762+0000 7f6906fb0640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690015dfd0 0x7f690015ae60 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f68f8009f40 tx=0x7f68f8004a40 comp rx=0 tx=0).stop 2024-04-24T23:50:01.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.762+0000 7f6906fb0640 1 -- 172.21.15.16:0/960908050 shutdown_connections 2024-04-24T23:50:01.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.762+0000 7f6906fb0640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f68e406be30 0x7f68e406e2f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.762+0000 7f6906fb0640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690015dfd0 0x7f690015ae60 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.762+0000 7f6906fb0640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f69000cefc0 0x7f69000cbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.763 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.762+0000 7f6906fb0640 1 --2- 172.21.15.16:0/960908050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f690006bc40 0x7f690014d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.762+0000 7f6906fb0640 1 -- 172.21.15.16:0/960908050 >> 172.21.15.16:0/960908050 conn(0x7f69000d3dc0 msgr2=0x7f69000d41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:01.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.762+0000 7f6906fb0640 1 -- 172.21.15.16:0/960908050 shutdown_connections 2024-04-24T23:50:01.764 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.762+0000 7f6906fb0640 1 -- 172.21.15.16:0/960908050 wait complete. 2024-04-24T23:50:01.773 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:50:01.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.875+0000 7f665c612640 1 Processor -- start 2024-04-24T23:50:01.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.875+0000 7f665c612640 1 -- start start 2024-04-24T23:50:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.876+0000 7f665c612640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f665415df20 0x7f6654071a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.876+0000 7f665c612640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f66540cfe50 0x7f6654071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.876+0000 7f665c612640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66540b6a70 con 0x7f66540d0530 2024-04-24T23:50:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.876+0000 7f665c612640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f66540cf460 con 0x7f665415df20 2024-04-24T23:50:01.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.876+0000 7f665c612640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f66540b6820 con 0x7f66540cfe50 2024-04-24T23:50:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.876+0000 7f665a3ae640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f665415df20 0x7f6654071a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.876+0000 7f6659bad640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f66540d0530 0x7f66540ad8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58972/0 (socket says 172.21.15.16:58972) 2024-04-24T23:50:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.876+0000 7f6659bad640 1 -- 172.21.15.16:0/4238141861 learned_addr learned my addr 172.21.15.16:0/4238141861 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.876+0000 7f665abaf640 1 --2- 172.21.15.16:0/4238141861 >> v2:172.21.15.156:3300/0 conn(0x7f66540cfe50 0x7f6654071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.876+0000 7f665a3ae640 1 -- 172.21.15.16:0/4238141861 >> v2:172.21.15.156:3300/0 conn(0x7f66540cfe50 msgr2=0x7f6654071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.876+0000 7f665a3ae640 1 --2- 172.21.15.16:0/4238141861 >> v2:172.21.15.156:3300/0 conn(0x7f66540cfe50 0x7f6654071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.876+0000 7f665a3ae640 1 -- 172.21.15.16:0/4238141861 >> v1:172.21.15.16:6789/0 conn(0x7f66540d0530 legacy=0x7f66540ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.876+0000 7f665a3ae640 1 -- 172.21.15.16:0/4238141861 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66540ce6a0 con 0x7f665415df20 2024-04-24T23:50:01.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.876+0000 7f665abaf640 1 --2- 172.21.15.16:0/4238141861 >> v2:172.21.15.156:3300/0 conn(0x7f66540cfe50 0x7f6654071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:01.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.877+0000 7f665a3ae640 1 --2- 172.21.15.16:0/4238141861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f665415df20 0x7f6654071a00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f66540b65c0 tx=0x7f664c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=964893df14a84128 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.877+0000 7f66593ac640 1 -- 172.21.15.16:0/4238141861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c037070 con 0x7f665415df20 2024-04-24T23:50:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.877+0000 7f66593ac640 1 -- 172.21.15.16:0/4238141861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f664c032440 con 0x7f665415df20 2024-04-24T23:50:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.877+0000 7f66593ac640 1 -- 172.21.15.16:0/4238141861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664c032810 con 0x7f665415df20 2024-04-24T23:50:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.878+0000 7f665c612640 1 -- 172.21.15.16:0/4238141861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f665415df20 msgr2=0x7f6654071a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.878+0000 7f665c612640 1 --2- 172.21.15.16:0/4238141861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f665415df20 0x7f6654071a00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f66540b65c0 tx=0x7f664c02a1a0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.878+0000 7f665c612640 1 -- 172.21.15.16:0/4238141861 shutdown_connections 2024-04-24T23:50:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.878+0000 7f665c612640 1 --2- 172.21.15.16:0/4238141861 >> v2:172.21.15.156:3300/0 conn(0x7f66540cfe50 0x7f6654071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.878+0000 7f665c612640 1 --2- 172.21.15.16:0/4238141861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f665415df20 0x7f6654071a00 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.878+0000 7f665c612640 1 -- 172.21.15.16:0/4238141861 >> 172.21.15.16:0/4238141861 conn(0x7f66540d0ac0 msgr2=0x7f66540d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.878+0000 7f665c612640 1 -- 172.21.15.16:0/4238141861 shutdown_connections 2024-04-24T23:50:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.878+0000 7f665c612640 1 -- 172.21.15.16:0/4238141861 wait complete. 2024-04-24T23:50:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.879+0000 7f665c612640 1 Processor -- start 2024-04-24T23:50:01.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.879+0000 7f665c612640 1 -- start start 2024-04-24T23:50:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665c612640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f66540cfe50 0x7f6654155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665c612640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66540d0530 0x7f6654150420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665c612640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f665415df20 0x7f66541f4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665c612640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f66540cef40 con 0x7f665415df20 2024-04-24T23:50:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665c612640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f66540cec60 con 0x7f66540d0530 2024-04-24T23:50:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665c612640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f66540b4030 con 0x7f66540cfe50 2024-04-24T23:50:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665abaf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f665415df20 0x7f66541f4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665abaf640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f665415df20 0x7f66541f4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56828/0 (socket says 172.21.15.16:56828) 2024-04-24T23:50:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665abaf640 1 -- 172.21.15.16:0/643123201 learned_addr learned my addr 172.21.15.16:0/643123201 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f6659bad640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66540d0530 0x7f6654150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665a3ae640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f66540cfe50 0x7f6654155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665abaf640 1 -- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f66540cfe50 msgr2=0x7f6654155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665abaf640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f66540cfe50 0x7f6654155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665abaf640 1 -- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66540d0530 msgr2=0x7f6654150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665abaf640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66540d0530 0x7f6654150420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:01.882 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665abaf640 1 -- 172.21.15.16:0/643123201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6654152f90 con 0x7f665415df20 2024-04-24T23:50:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f665a3ae640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f66540cfe50 0x7f6654155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.880+0000 7f6659bad640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66540d0530 0x7f6654150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.881+0000 7f665abaf640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f665415df20 0x7f66541f4c10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6644009f70 tx=0x7f664400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.881+0000 7f6652ffd640 1 -- 172.21.15.16:0/643123201 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f664401a070 con 0x7f665415df20 2024-04-24T23:50:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.881+0000 7f6652ffd640 1 -- 172.21.15.16:0/643123201 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6644003bb0 con 0x7f665415df20 2024-04-24T23:50:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.881+0000 7f665c612640 1 -- 172.21.15.16:0/643123201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66541561a0 con 0x7f665415df20 2024-04-24T23:50:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.881+0000 7f6652ffd640 1 -- 172.21.15.16:0/643123201 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6644004030 con 0x7f665415df20 2024-04-24T23:50:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.881+0000 7f665c612640 1 -- 172.21.15.16:0/643123201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66541f4410 con 0x7f665415df20 2024-04-24T23:50:01.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.882+0000 7f665c612640 1 -- 172.21.15.16:0/643123201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f665406f8f0 con 0x7f665415df20 2024-04-24T23:50:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.882+0000 7f6652ffd640 1 -- 172.21.15.16:0/643123201 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f664401d070 con 0x7f665415df20 2024-04-24T23:50:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.883+0000 7f6652ffd640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f664006be30 0x7f664006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.883+0000 7f6652ffd640 1 -- 172.21.15.16:0/643123201 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f664408b570 con 0x7f665415df20 2024-04-24T23:50:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.883+0000 7f665a3ae640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f664006be30 0x7f664006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:01.884 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.884+0000 7f665a3ae640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f664006be30 0x7f664006e2f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f664c034000 tx=0x7f664c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:01.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:01.887+0000 7f6652ffd640 1 -- 172.21.15.16:0/643123201 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6654152f90 con 0x7f665415df20 2024-04-24T23:50:02.084 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.084+0000 7f665c612640 1 -- 172.21.15.16:0/643123201 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6654072b40 con 0x7f665415df20 2024-04-24T23:50:02.085 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.084+0000 7f6652ffd640 1 -- 172.21.15.16:0/643123201 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7f6644010070 con 0x7f665415df20 2024-04-24T23:50:02.085 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-04-24T23:50:02.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.086+0000 7f665c612640 1 -- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f664006be30 msgr2=0x7f664006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.086+0000 7f665c612640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f664006be30 0x7f664006e2f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f664c034000 tx=0x7f664c02e000 comp rx=0 tx=0).stop 2024-04-24T23:50:02.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.086+0000 7f665c612640 1 -- 172.21.15.16:0/643123201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f665415df20 msgr2=0x7f66541f4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.086+0000 7f665c612640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f665415df20 0x7f66541f4c10 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6644009f70 tx=0x7f664400d560 comp rx=0 tx=0).stop 2024-04-24T23:50:02.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.086+0000 7f665c612640 1 -- 172.21.15.16:0/643123201 shutdown_connections 2024-04-24T23:50:02.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.086+0000 7f665c612640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f664006be30 0x7f664006e2f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.086+0000 7f665c612640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f665415df20 0x7f66541f4c10 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.086+0000 7f665c612640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f66540d0530 0x7f6654150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.086+0000 7f665c612640 1 --2- 172.21.15.16:0/643123201 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f66540cfe50 0x7f6654155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.086+0000 7f665c612640 1 -- 172.21.15.16:0/643123201 >> 172.21.15.16:0/643123201 conn(0x7f66540d0ac0 msgr2=0x7f66540aea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:02.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.086+0000 7f665c612640 1 -- 172.21.15.16:0/643123201 shutdown_connections 2024-04-24T23:50:02.088 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.086+0000 7f665c612640 1 -- 172.21.15.16:0/643123201 wait complete. 2024-04-24T23:50:02.097 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:50:02.097 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:50:02.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:50:02.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.218+0000 7ff4d22d4640 1 Processor -- start 2024-04-24T23:50:02.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.218+0000 7ff4d22d4640 1 -- start start 2024-04-24T23:50:02.221 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.219+0000 7ff4d22d4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4cc15c250 0x7ff4cc072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.219+0000 7ff4d22d4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff4cc0727d0 0x7ff4cc0ced40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.219+0000 7ff4d22d4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4cc06cee0 con 0x7ff4cc06cb00 2024-04-24T23:50:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.219+0000 7ff4d22d4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ff4cc0ce760 con 0x7ff4cc0727d0 2024-04-24T23:50:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.219+0000 7ff4d22d4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff4cc15c8b0 con 0x7ff4cc15c250 2024-04-24T23:50:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.220+0000 7ff4cb7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4cc15c250 0x7ff4cc072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.220+0000 7ff4d0871640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff4cc0727d0 0x7ff4cc0ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.220+0000 7ff4cb7fe640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4cc15c250 0x7ff4cc072290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:56558/0 (socket says 172.21.15.16:56558) 2024-04-24T23:50:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.220+0000 7ff4cb7fe640 1 -- 172.21.15.16:0/1116585434 learned_addr learned my addr 172.21.15.16:0/1116585434 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:02.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.220+0000 7ff4cbfff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7ff4cc06cb00 0x7ff4cc0ccc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:40770/0 (socket says 172.21.15.16:40770) 2024-04-24T23:50:02.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.220+0000 7ff4cb7fe640 1 -- 172.21.15.16:0/1116585434 >> v2:172.21.15.156:3300/0 conn(0x7ff4cc0727d0 msgr2=0x7ff4cc0ced40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:02.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.220+0000 7ff4cb7fe640 1 --2- 172.21.15.16:0/1116585434 >> v2:172.21.15.156:3300/0 conn(0x7ff4cc0727d0 0x7ff4cc0ced40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.220+0000 7ff4cb7fe640 1 -- 172.21.15.16:0/1116585434 >> v1:172.21.15.156:6789/0 conn(0x7ff4cc06cb00 legacy=0x7ff4cc0ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.220+0000 7ff4cb7fe640 1 -- 172.21.15.16:0/1116585434 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4cc15cb90 con 0x7ff4cc15c250 2024-04-24T23:50:02.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.221+0000 7ff4cb7fe640 1 --2- 172.21.15.16:0/1116585434 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4cc15c250 0x7ff4cc072290 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009870 tx=0x7ff4bc02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7aef4f387eb8962d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:02.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.221+0000 7ff4caffd640 1 -- 172.21.15.16:0/1116585434 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc037070 con 0x7ff4cc15c250 2024-04-24T23:50:02.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.221+0000 7ff4caffd640 1 -- 172.21.15.16:0/1116585434 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4bc032440 con 0x7ff4cc15c250 2024-04-24T23:50:02.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.221+0000 7ff4caffd640 1 -- 172.21.15.16:0/1116585434 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc032810 con 0x7ff4cc15c250 2024-04-24T23:50:02.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.221+0000 7ff4d22d4640 1 -- 172.21.15.16:0/1116585434 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4cc15c250 msgr2=0x7ff4cc072290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.221+0000 7ff4d22d4640 1 --2- 172.21.15.16:0/1116585434 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4cc15c250 0x7ff4cc072290 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc009870 tx=0x7ff4bc02a270 comp rx=0 tx=0).stop 2024-04-24T23:50:02.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.221+0000 7ff4d22d4640 1 -- 172.21.15.16:0/1116585434 shutdown_connections 2024-04-24T23:50:02.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.221+0000 7ff4d22d4640 1 --2- 172.21.15.16:0/1116585434 >> v2:172.21.15.156:3300/0 conn(0x7ff4cc0727d0 0x7ff4cc0ced40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.221+0000 7ff4d22d4640 1 --2- 172.21.15.16:0/1116585434 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4cc15c250 0x7ff4cc072290 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.221+0000 7ff4d22d4640 1 -- 172.21.15.16:0/1116585434 >> 172.21.15.16:0/1116585434 conn(0x7ff4cc0d0260 msgr2=0x7ff4cc0d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:02.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.221+0000 7ff4d22d4640 1 -- 172.21.15.16:0/1116585434 shutdown_connections 2024-04-24T23:50:02.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.221+0000 7ff4d22d4640 1 -- 172.21.15.16:0/1116585434 wait complete. 2024-04-24T23:50:02.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.223+0000 7ff4d22d4640 1 Processor -- start 2024-04-24T23:50:02.228 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.223+0000 7ff4d22d4640 1 -- start start 2024-04-24T23:50:02.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4d22d4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4cc06cb00 0x7ff4cc149240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4d22d4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4cc0727d0 0x7ff4cc147f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4d22d4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4cc15c250 0x7ff4cc0b3e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4d22d4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff4cc06b0e0 con 0x7ff4cc06cb00 2024-04-24T23:50:02.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4d22d4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff4cc15d150 con 0x7ff4cc15c250 2024-04-24T23:50:02.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4d22d4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff4cc0cf280 con 0x7ff4cc0727d0 2024-04-24T23:50:02.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4cbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4cc06cb00 0x7ff4cc149240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4cbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4cc06cb00 0x7ff4cc149240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56836/0 (socket says 172.21.15.16:56836) 2024-04-24T23:50:02.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4cbfff640 1 -- 172.21.15.16:0/176564134 learned_addr learned my addr 172.21.15.16:0/176564134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:02.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4cb7fe640 1 --2- 172.21.15.16:0/176564134 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4cc0727d0 0x7ff4cc147f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4d0871640 1 --2- 172.21.15.16:0/176564134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4cc15c250 0x7ff4cc0b3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4cbfff640 1 -- 172.21.15.16:0/176564134 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4cc0727d0 msgr2=0x7ff4cc147f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4cbfff640 1 --2- 172.21.15.16:0/176564134 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4cc0727d0 0x7ff4cc147f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4cbfff640 1 -- 172.21.15.16:0/176564134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4cc15c250 msgr2=0x7ff4cc0b3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4cbfff640 1 --2- 172.21.15.16:0/176564134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4cc15c250 0x7ff4cc0b3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4cbfff640 1 -- 172.21.15.16:0/176564134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4cc1484d0 con 0x7ff4cc06cb00 2024-04-24T23:50:02.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4cbfff640 1 --2- 172.21.15.16:0/176564134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4cc06cb00 0x7ff4cc149240 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000c430 tx=0x7ff4c000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:02.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4c8ff9640 1 -- 172.21.15.16:0/176564134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c00074e0 con 0x7ff4cc06cb00 2024-04-24T23:50:02.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4c8ff9640 1 -- 172.21.15.16:0/176564134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c0007680 con 0x7ff4cc06cb00 2024-04-24T23:50:02.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4d22d4640 1 -- 172.21.15.16:0/176564134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4cc0b48e0 con 0x7ff4cc06cb00 2024-04-24T23:50:02.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4c8ff9640 1 -- 172.21.15.16:0/176564134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0004a00 con 0x7ff4cc06cb00 2024-04-24T23:50:02.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.224+0000 7ff4d22d4640 1 -- 172.21.15.16:0/176564134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4cc148a40 con 0x7ff4cc06cb00 2024-04-24T23:50:02.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.225+0000 7ff4d22d4640 1 -- 172.21.15.16:0/176564134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4cc04f5e0 con 0x7ff4cc06cb00 2024-04-24T23:50:02.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.226+0000 7ff4c8ff9640 1 -- 172.21.15.16:0/176564134 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff4c001a020 con 0x7ff4cc06cb00 2024-04-24T23:50:02.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.226+0000 7ff4c8ff9640 1 --2- 172.21.15.16:0/176564134 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff4b806be30 0x7ff4b806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.226+0000 7ff4c8ff9640 1 -- 172.21.15.16:0/176564134 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7ff4c008aa70 con 0x7ff4cc06cb00 2024-04-24T23:50:02.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.227+0000 7ff4cb7fe640 1 --2- 172.21.15.16:0/176564134 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff4b806be30 0x7ff4b806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.227+0000 7ff4cb7fe640 1 --2- 172.21.15.16:0/176564134 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff4b806be30 0x7ff4b806e2f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc034000 tx=0x7ff4bc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:02.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.231+0000 7ff4c8ff9640 1 -- 172.21.15.16:0/176564134 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff4c0059020 con 0x7ff4cc06cb00 2024-04-24T23:50:02.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.416+0000 7ff4d22d4640 1 -- 172.21.15.16:0/176564134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff4cc072e90 con 0x7ff4cc06cb00 2024-04-24T23:50:02.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.418+0000 7ff4c8ff9640 1 -- 172.21.15.16:0/176564134 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7ff4c0058920 con 0x7ff4cc06cb00 2024-04-24T23:50:02.418 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:02.418 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:50:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.420+0000 7ff4d22d4640 1 -- 172.21.15.16:0/176564134 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff4b806be30 msgr2=0x7ff4b806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.420+0000 7ff4d22d4640 1 --2- 172.21.15.16:0/176564134 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff4b806be30 0x7ff4b806e2f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc034000 tx=0x7ff4bc02e000 comp rx=0 tx=0).stop 2024-04-24T23:50:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.420+0000 7ff4d22d4640 1 -- 172.21.15.16:0/176564134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4cc06cb00 msgr2=0x7ff4cc149240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.420+0000 7ff4d22d4640 1 --2- 172.21.15.16:0/176564134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4cc06cb00 0x7ff4cc149240 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000c430 tx=0x7ff4c000c8f0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.420+0000 7ff4d22d4640 1 -- 172.21.15.16:0/176564134 shutdown_connections 2024-04-24T23:50:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.420+0000 7ff4d22d4640 1 --2- 172.21.15.16:0/176564134 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff4b806be30 0x7ff4b806e2f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.420+0000 7ff4d22d4640 1 --2- 172.21.15.16:0/176564134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4cc15c250 0x7ff4cc0b3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.420+0000 7ff4d22d4640 1 --2- 172.21.15.16:0/176564134 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4cc0727d0 0x7ff4cc147f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.420+0000 7ff4d22d4640 1 --2- 172.21.15.16:0/176564134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4cc06cb00 0x7ff4cc149240 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.420+0000 7ff4d22d4640 1 -- 172.21.15.16:0/176564134 >> 172.21.15.16:0/176564134 conn(0x7ff4cc0d0260 msgr2=0x7ff4cc0b13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.420+0000 7ff4d22d4640 1 -- 172.21.15.16:0/176564134 shutdown_connections 2024-04-24T23:50:02.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.420+0000 7ff4d22d4640 1 -- 172.21.15.16:0/176564134 wait complete. 2024-04-24T23:50:02.431 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:50:02.432 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:50:02.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.536+0000 7f75ee6f5640 1 Processor -- start 2024-04-24T23:50:02.536 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.536+0000 7f75ee6f5640 1 -- start start 2024-04-24T23:50:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.537+0000 7f75ee6f5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75e81455f0 0x7f75e806b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.537+0000 7f75ee6f5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f75e80cee70 0x7f75e806ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.537+0000 7f75ee6f5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75e815f9c0 con 0x7f75e806b620 2024-04-24T23:50:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.537+0000 7f75ee6f5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f75e81605d0 con 0x7f75e81455f0 2024-04-24T23:50:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.537+0000 7f75ee6f5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f75e815fca0 con 0x7f75e80cee70 2024-04-24T23:50:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.537+0000 7f75e7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75e81455f0 0x7f75e806b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.537+0000 7f75e7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75e81455f0 0x7f75e806b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56850/0 (socket says 172.21.15.16:56850) 2024-04-24T23:50:02.538 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.537+0000 7f75e7fff640 1 -- 172.21.15.16:0/208193610 learned_addr learned my addr 172.21.15.16:0/208193610 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.538+0000 7f75e7fff640 1 -- 172.21.15.16:0/208193610 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f75e80cee70 msgr2=0x7f75e806ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.538+0000 7f75e7fff640 1 --2- 172.21.15.16:0/208193610 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f75e80cee70 0x7f75e806ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.538+0000 7f75e7fff640 1 -- 172.21.15.16:0/208193610 >> v1:172.21.15.156:6789/0 conn(0x7f75e806b620 legacy=0x7f75e80ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.538+0000 7f75e7fff640 1 -- 172.21.15.16:0/208193610 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e815ff80 con 0x7f75e81455f0 2024-04-24T23:50:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.538+0000 7f75e7fff640 1 --2- 172.21.15.16:0/208193610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e81455f0 0x7f75e806b0e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f75dc009870 tx=0x7f75dc02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7364f6ff0595a152 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.538+0000 7f75e6ffd640 1 -- 172.21.15.16:0/208193610 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc037070 con 0x7f75e81455f0 2024-04-24T23:50:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.538+0000 7f75e6ffd640 1 -- 172.21.15.16:0/208193610 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75dc032440 con 0x7f75e81455f0 2024-04-24T23:50:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.538+0000 7f75e6ffd640 1 -- 172.21.15.16:0/208193610 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc032810 con 0x7f75e81455f0 2024-04-24T23:50:02.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.539+0000 7f75ee6f5640 1 -- 172.21.15.16:0/208193610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e81455f0 msgr2=0x7f75e806b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.539+0000 7f75ee6f5640 1 --2- 172.21.15.16:0/208193610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e81455f0 0x7f75e806b0e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f75dc009870 tx=0x7f75dc02a180 comp rx=0 tx=0).stop 2024-04-24T23:50:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.539+0000 7f75ee6f5640 1 -- 172.21.15.16:0/208193610 shutdown_connections 2024-04-24T23:50:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.539+0000 7f75ee6f5640 1 --2- 172.21.15.16:0/208193610 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f75e80cee70 0x7f75e806ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.539+0000 7f75ee6f5640 1 --2- 172.21.15.16:0/208193610 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e81455f0 0x7f75e806b0e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.539+0000 7f75ee6f5640 1 -- 172.21.15.16:0/208193610 >> 172.21.15.16:0/208193610 conn(0x7f75e80cfc50 msgr2=0x7f75e80d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.539+0000 7f75ee6f5640 1 -- 172.21.15.16:0/208193610 shutdown_connections 2024-04-24T23:50:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.539+0000 7f75ee6f5640 1 -- 172.21.15.16:0/208193610 wait complete. 2024-04-24T23:50:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.540+0000 7f75ee6f5640 1 Processor -- start 2024-04-24T23:50:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.540+0000 7f75ee6f5640 1 -- start start 2024-04-24T23:50:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.541+0000 7f75ee6f5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f75e806b620 0x7f75e815b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.541+0000 7f75ee6f5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e80cee70 0x7f75e813bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.541+0000 7f75ee6f5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75e81455f0 0x7f75e80b45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.541+0000 7f75ee6f5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f75e815d0a0 con 0x7f75e80cee70 2024-04-24T23:50:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.541+0000 7f75ee6f5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f75e81459d0 con 0x7f75e81455f0 2024-04-24T23:50:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.541+0000 7f75ee6f5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f75e815d220 con 0x7f75e806b620 2024-04-24T23:50:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.541+0000 7f75e77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e80cee70 0x7f75e813bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.541+0000 7f75e77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e80cee70 0x7f75e813bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56858/0 (socket says 172.21.15.16:56858) 2024-04-24T23:50:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.541+0000 7f75e77fe640 1 -- 172.21.15.16:0/540707280 learned_addr learned my addr 172.21.15.16:0/540707280 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.541+0000 7f75e7fff640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f75e806b620 0x7f75e815b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.541+0000 7f75ecc92640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75e81455f0 0x7f75e80b45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.542+0000 7f75e77fe640 1 -- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f75e806b620 msgr2=0x7f75e815b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.542+0000 7f75e77fe640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f75e806b620 0x7f75e815b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.542+0000 7f75e77fe640 1 -- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75e81455f0 msgr2=0x7f75e80b45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.542+0000 7f75e77fe640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75e81455f0 0x7f75e80b45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.542+0000 7f75e77fe640 1 -- 172.21.15.16:0/540707280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75e813c4d0 con 0x7f75e80cee70 2024-04-24T23:50:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.542+0000 7f75ecc92640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75e81455f0 0x7f75e80b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:02.544 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.542+0000 7f75e77fe640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e80cee70 0x7f75e813bb60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f75d800b3f0 tx=0x7f75d800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.542+0000 7f75e4ff9640 1 -- 172.21.15.16:0/540707280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d8015020 con 0x7f75e80cee70 2024-04-24T23:50:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.542+0000 7f75e4ff9640 1 -- 172.21.15.16:0/540707280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75d8011830 con 0x7f75e80cee70 2024-04-24T23:50:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.542+0000 7f75ee6f5640 1 -- 172.21.15.16:0/540707280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75e813c7b0 con 0x7f75e80cee70 2024-04-24T23:50:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.542+0000 7f75e4ff9640 1 -- 172.21.15.16:0/540707280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75d8011c20 con 0x7f75e80cee70 2024-04-24T23:50:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.542+0000 7f75ee6f5640 1 -- 172.21.15.16:0/540707280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75e813cd70 con 0x7f75e80cee70 2024-04-24T23:50:02.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.543+0000 7f75ee6f5640 1 -- 172.21.15.16:0/540707280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75e8068f90 con 0x7f75e80cee70 2024-04-24T23:50:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.544+0000 7f75e4ff9640 1 -- 172.21.15.16:0/540707280 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f75d8018020 con 0x7f75e80cee70 2024-04-24T23:50:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.544+0000 7f75e4ff9640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f75d406be60 0x7f75d406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.544+0000 7f75e4ff9640 1 -- 172.21.15.16:0/540707280 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f75d808adc0 con 0x7f75e80cee70 2024-04-24T23:50:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.544+0000 7f75e7fff640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f75d406be60 0x7f75d406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.545+0000 7f75e7fff640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f75d406be60 0x7f75d406e320 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f75dc034000 tx=0x7f75dc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:02.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.549+0000 7f75e4ff9640 1 -- 172.21.15.16:0/540707280 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f75d801f030 con 0x7f75e80cee70 2024-04-24T23:50:02.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.731+0000 7f75ee6f5640 1 -- 172.21.15.16:0/540707280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f75e815bdd0 con 0x7f75e80cee70 2024-04-24T23:50:02.732 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.732+0000 7f75e4ff9640 1 -- 172.21.15.16:0/540707280 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f75d80593e0 con 0x7f75e80cee70 2024-04-24T23:50:02.732 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:50:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.734+0000 7f75ee6f5640 1 -- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f75d406be60 msgr2=0x7f75d406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.734+0000 7f75ee6f5640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f75d406be60 0x7f75d406e320 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f75dc034000 tx=0x7f75dc02e000 comp rx=0 tx=0).stop 2024-04-24T23:50:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.734+0000 7f75ee6f5640 1 -- 172.21.15.16:0/540707280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e80cee70 msgr2=0x7f75e813bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.734+0000 7f75ee6f5640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e80cee70 0x7f75e813bb60 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f75d800b3f0 tx=0x7f75d800b8b0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.734+0000 7f75ee6f5640 1 -- 172.21.15.16:0/540707280 shutdown_connections 2024-04-24T23:50:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.734+0000 7f75ee6f5640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f75d406be60 0x7f75d406e320 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.734+0000 7f75ee6f5640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f75e81455f0 0x7f75e80b45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.734+0000 7f75ee6f5640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75e80cee70 0x7f75e813bb60 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.734+0000 7f75ee6f5640 1 --2- 172.21.15.16:0/540707280 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f75e806b620 0x7f75e815b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.734+0000 7f75ee6f5640 1 -- 172.21.15.16:0/540707280 >> 172.21.15.16:0/540707280 conn(0x7f75e80cfc50 msgr2=0x7f75e815d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.734+0000 7f75ee6f5640 1 -- 172.21.15.16:0/540707280 shutdown_connections 2024-04-24T23:50:02.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.734+0000 7f75ee6f5640 1 -- 172.21.15.16:0/540707280 wait complete. 2024-04-24T23:50:02.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:50:02.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.850+0000 7fe758c38640 1 Processor -- start 2024-04-24T23:50:02.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.850+0000 7fe758c38640 1 -- start start 2024-04-24T23:50:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.851+0000 7fe758c38640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe75415dfd0 0x7fe75406b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.851+0000 7fe758c38640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe7540b8060 0x7fe75406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.851+0000 7fe758c38640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7540d24d0 con 0x7fe75406bc40 2024-04-24T23:50:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.851+0000 7fe758c38640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fe7540d27b0 con 0x7fe75415dfd0 2024-04-24T23:50:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.851+0000 7fe758c38640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fe7540d2a90 con 0x7fe7540b8060 2024-04-24T23:50:02.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.851+0000 7fe75259c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe75415dfd0 0x7fe75406b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.851+0000 7fe75259c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe75415dfd0 0x7fe75406b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56874/0 (socket says 172.21.15.16:56874) 2024-04-24T23:50:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.851+0000 7fe75259c640 1 -- 172.21.15.16:0/1325336881 learned_addr learned my addr 172.21.15.16:0/1325336881 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.851+0000 7fe752d9d640 1 --2- 172.21.15.16:0/1325336881 >> v2:172.21.15.156:3300/0 conn(0x7fe7540b8060 0x7fe75406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.851+0000 7fe75259c640 1 -- 172.21.15.16:0/1325336881 >> v2:172.21.15.156:3300/0 conn(0x7fe7540b8060 msgr2=0x7fe75406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.851+0000 7fe75259c640 1 --2- 172.21.15.16:0/1325336881 >> v2:172.21.15.156:3300/0 conn(0x7fe7540b8060 0x7fe75406c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.851+0000 7fe75259c640 1 -- 172.21.15.16:0/1325336881 >> v1:172.21.15.156:6789/0 conn(0x7fe75406bc40 legacy=0x7fe7540cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.851+0000 7fe75259c640 1 -- 172.21.15.16:0/1325336881 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7540d2d70 con 0x7fe75415dfd0 2024-04-24T23:50:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.852+0000 7fe75259c640 1 --2- 172.21.15.16:0/1325336881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75415dfd0 0x7fe75406b2f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe7540b6ca0 tx=0x7fe74c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d247434f4020c0c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.852+0000 7fe75159a640 1 -- 172.21.15.16:0/1325336881 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c037070 con 0x7fe75415dfd0 2024-04-24T23:50:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.852+0000 7fe75159a640 1 -- 172.21.15.16:0/1325336881 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe74c032440 con 0x7fe75415dfd0 2024-04-24T23:50:02.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.852+0000 7fe75159a640 1 -- 172.21.15.16:0/1325336881 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe74c032810 con 0x7fe75415dfd0 2024-04-24T23:50:02.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.853+0000 7fe758c38640 1 -- 172.21.15.16:0/1325336881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75415dfd0 msgr2=0x7fe75406b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.853+0000 7fe758c38640 1 --2- 172.21.15.16:0/1325336881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75415dfd0 0x7fe75406b2f0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe7540b6ca0 tx=0x7fe74c02a1e0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.853+0000 7fe758c38640 1 -- 172.21.15.16:0/1325336881 shutdown_connections 2024-04-24T23:50:02.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.853+0000 7fe758c38640 1 --2- 172.21.15.16:0/1325336881 >> v2:172.21.15.156:3300/0 conn(0x7fe7540b8060 0x7fe75406c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.853+0000 7fe758c38640 1 --2- 172.21.15.16:0/1325336881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75415dfd0 0x7fe75406b2f0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.853+0000 7fe758c38640 1 -- 172.21.15.16:0/1325336881 >> 172.21.15.16:0/1325336881 conn(0x7fe7540d3dc0 msgr2=0x7fe7540d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:02.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.853+0000 7fe758c38640 1 -- 172.21.15.16:0/1325336881 shutdown_connections 2024-04-24T23:50:02.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.853+0000 7fe758c38640 1 -- 172.21.15.16:0/1325336881 wait complete. 2024-04-24T23:50:02.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.854+0000 7fe758c38640 1 Processor -- start 2024-04-24T23:50:02.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.854+0000 7fe758c38640 1 -- start start 2024-04-24T23:50:02.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.855+0000 7fe758c38640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe75406bc40 0x7fe7541f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.855+0000 7fe758c38640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7540b8060 0x7fe7541e66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.855+0000 7fe758c38640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe75415dfd0 0x7fe75414d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.855+0000 7fe758c38640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe75406b830 con 0x7fe7540b8060 2024-04-24T23:50:02.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.855+0000 7fe758c38640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe7540cead0 con 0x7fe75406bc40 2024-04-24T23:50:02.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.855+0000 7fe758c38640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe75406b9b0 con 0x7fe75415dfd0 2024-04-24T23:50:02.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.855+0000 7fe751d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7540b8060 0x7fe7541e66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.855+0000 7fe751d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7540b8060 0x7fe7541e66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56880/0 (socket says 172.21.15.16:56880) 2024-04-24T23:50:02.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.855+0000 7fe751d9b640 1 -- 172.21.15.16:0/3020416218 learned_addr learned my addr 172.21.15.16:0/3020416218 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:02.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.856+0000 7fe75259c640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe75406bc40 0x7fe7541f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.856+0000 7fe752d9d640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe75415dfd0 0x7fe75414d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.856+0000 7fe751d9b640 1 -- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe75415dfd0 msgr2=0x7fe75414d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.856+0000 7fe751d9b640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe75415dfd0 0x7fe75414d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.856+0000 7fe751d9b640 1 -- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe75406bc40 msgr2=0x7fe7541f74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:02.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.856+0000 7fe751d9b640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe75406bc40 0x7fe7541f74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:02.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.856+0000 7fe751d9b640 1 -- 172.21.15.16:0/3020416218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7541f83e0 con 0x7fe7540b8060 2024-04-24T23:50:02.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.856+0000 7fe75259c640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe75406bc40 0x7fe7541f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:02.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.856+0000 7fe751d9b640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7540b8060 0x7fe7541e66b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c400 tx=0x7fe74800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:02.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.856+0000 7fe742ffd640 1 -- 172.21.15.16:0/3020416218 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe748017070 con 0x7fe7540b8060 2024-04-24T23:50:02.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.856+0000 7fe742ffd640 1 -- 172.21.15.16:0/3020416218 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7480052d0 con 0x7fe7540b8060 2024-04-24T23:50:02.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.856+0000 7fe758c38640 1 -- 172.21.15.16:0/3020416218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7541f7b40 con 0x7fe7540b8060 2024-04-24T23:50:02.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.856+0000 7fe742ffd640 1 -- 172.21.15.16:0/3020416218 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe748003830 con 0x7fe7540b8060 2024-04-24T23:50:02.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.856+0000 7fe758c38640 1 -- 172.21.15.16:0/3020416218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7541f8100 con 0x7fe7540b8060 2024-04-24T23:50:02.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.857+0000 7fe758c38640 1 -- 172.21.15.16:0/3020416218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe754068f90 con 0x7fe7540b8060 2024-04-24T23:50:02.877 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.858+0000 7fe742ffd640 1 -- 172.21.15.16:0/3020416218 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe74801a070 con 0x7fe7540b8060 2024-04-24T23:50:02.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.858+0000 7fe742ffd640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe73406be30 0x7fe73406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:02.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.858+0000 7fe742ffd640 1 -- 172.21.15.16:0/3020416218 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe74808a530 con 0x7fe7540b8060 2024-04-24T23:50:02.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.858+0000 7fe75259c640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe73406be30 0x7fe73406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:02.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.859+0000 7fe75259c640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe73406be30 0x7fe73406e2f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe74c034000 tx=0x7fe74c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:02.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:02.862+0000 7fe742ffd640 1 -- 172.21.15.16:0/3020416218 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe748058b70 con 0x7fe7540b8060 2024-04-24T23:50:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.043+0000 7fe758c38640 1 -- 172.21.15.16:0/3020416218 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe7540b45d0 con 0x7fe7540b8060 2024-04-24T23:50:03.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.044+0000 7fe742ffd640 1 -- 172.21.15.16:0/3020416218 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fe748058470 con 0x7fe7540b8060 2024-04-24T23:50:03.044 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:50:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.045+0000 7fe758c38640 1 -- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe73406be30 msgr2=0x7fe73406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.045+0000 7fe758c38640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe73406be30 0x7fe73406e2f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe74c034000 tx=0x7fe74c02e000 comp rx=0 tx=0).stop 2024-04-24T23:50:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.045+0000 7fe758c38640 1 -- 172.21.15.16:0/3020416218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7540b8060 msgr2=0x7fe7541e66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.045+0000 7fe758c38640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7540b8060 0x7fe7541e66b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c400 tx=0x7fe74800c8c0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.045+0000 7fe758c38640 1 -- 172.21.15.16:0/3020416218 shutdown_connections 2024-04-24T23:50:03.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.045+0000 7fe758c38640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe73406be30 0x7fe73406e2f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.045+0000 7fe758c38640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe75415dfd0 0x7fe75414d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.045+0000 7fe758c38640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe7540b8060 0x7fe7541e66b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.045+0000 7fe758c38640 1 --2- 172.21.15.16:0/3020416218 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe75406bc40 0x7fe7541f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.045+0000 7fe758c38640 1 -- 172.21.15.16:0/3020416218 >> 172.21.15.16:0/3020416218 conn(0x7fe7540d3dc0 msgr2=0x7fe7540b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:03.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.046+0000 7fe758c38640 1 -- 172.21.15.16:0/3020416218 shutdown_connections 2024-04-24T23:50:03.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.046+0000 7fe758c38640 1 -- 172.21.15.16:0/3020416218 wait complete. 2024-04-24T23:50:03.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:50:03.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.160+0000 7f42f9f42640 1 Processor -- start 2024-04-24T23:50:03.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.160+0000 7f42f9f42640 1 -- start start 2024-04-24T23:50:03.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.161+0000 7f42f9f42640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f42f415dfd0 0x7f42f406b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.161+0000 7f42f9f42640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f42f40b8060 0x7f42f406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.161+0000 7f42f9f42640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42f40d24d0 con 0x7f42f406bc40 2024-04-24T23:50:03.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.161+0000 7f42f9f42640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f42f40d27b0 con 0x7f42f415dfd0 2024-04-24T23:50:03.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.161+0000 7f42f9f42640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f42f40d2a90 con 0x7f42f40b8060 2024-04-24T23:50:03.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.161+0000 7f42f37fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f42f415dfd0 0x7f42f406b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.161+0000 7f42f2ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f42f406bc40 0x7f42f40cefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58978/0 (socket says 172.21.15.16:58978) 2024-04-24T23:50:03.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.161+0000 7f42f2ffd640 1 -- 172.21.15.16:0/1917210373 learned_addr learned my addr 172.21.15.16:0/1917210373 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:03.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.162+0000 7f42f3fff640 1 --2- 172.21.15.16:0/1917210373 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f42f40b8060 0x7f42f406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.162+0000 7f42f37fe640 1 -- 172.21.15.16:0/1917210373 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f42f40b8060 msgr2=0x7f42f406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:03.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.162+0000 7f42f37fe640 1 --2- 172.21.15.16:0/1917210373 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f42f40b8060 0x7f42f406c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.162+0000 7f42f37fe640 1 -- 172.21.15.16:0/1917210373 >> v1:172.21.15.16:6789/0 conn(0x7f42f406bc40 legacy=0x7f42f40cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.162+0000 7f42f37fe640 1 -- 172.21.15.16:0/1917210373 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f40d2d70 con 0x7f42f415dfd0 2024-04-24T23:50:03.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.162+0000 7f42f27fc640 1 -- 172.21.15.16:0/1917210373 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2794762959 0 0) 0x7f42f40d24d0 con 0x7f42f406bc40 2024-04-24T23:50:03.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.162+0000 7f42f3fff640 1 --2- 172.21.15.16:0/1917210373 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f42f40b8060 0x7f42f406c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:03.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.162+0000 7f42f37fe640 1 --2- 172.21.15.16:0/1917210373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f415dfd0 0x7f42f406b2f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f42e40097a0 tx=0x7f42e402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2459615a3ee2d5c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:03.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.163+0000 7f42f27fc640 1 -- 172.21.15.16:0/1917210373 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e4037070 con 0x7f42f415dfd0 2024-04-24T23:50:03.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.163+0000 7f42f27fc640 1 -- 172.21.15.16:0/1917210373 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42e402ae50 con 0x7f42f415dfd0 2024-04-24T23:50:03.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.163+0000 7f42f27fc640 1 -- 172.21.15.16:0/1917210373 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42e4032660 con 0x7f42f415dfd0 2024-04-24T23:50:03.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.163+0000 7f42f9f42640 1 -- 172.21.15.16:0/1917210373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f415dfd0 msgr2=0x7f42f406b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.163+0000 7f42f9f42640 1 --2- 172.21.15.16:0/1917210373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f415dfd0 0x7f42f406b2f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f42e40097a0 tx=0x7f42e402a100 comp rx=0 tx=0).stop 2024-04-24T23:50:03.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.163+0000 7f42f9f42640 1 -- 172.21.15.16:0/1917210373 shutdown_connections 2024-04-24T23:50:03.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.163+0000 7f42f9f42640 1 --2- 172.21.15.16:0/1917210373 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f42f40b8060 0x7f42f406c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.163+0000 7f42f9f42640 1 --2- 172.21.15.16:0/1917210373 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f415dfd0 0x7f42f406b2f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.163+0000 7f42f9f42640 1 -- 172.21.15.16:0/1917210373 >> 172.21.15.16:0/1917210373 conn(0x7f42f40d3dc0 msgr2=0x7f42f40d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:03.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.163+0000 7f42f9f42640 1 -- 172.21.15.16:0/1917210373 shutdown_connections 2024-04-24T23:50:03.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.163+0000 7f42f9f42640 1 -- 172.21.15.16:0/1917210373 wait complete. 2024-04-24T23:50:03.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.164+0000 7f42f9f42640 1 Processor -- start 2024-04-24T23:50:03.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.165+0000 7f42f9f42640 1 -- start start 2024-04-24T23:50:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.165+0000 7f42f9f42640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42f406bc40 0x7f42f40cd620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.165+0000 7f42f9f42640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f42f40b8060 0x7f42f415a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.165+0000 7f42f9f42640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f415dfd0 0x7f42f415acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.165+0000 7f42f9f42640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f42f406b830 con 0x7f42f415dfd0 2024-04-24T23:50:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.165+0000 7f42f9f42640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f42f40cead0 con 0x7f42f406bc40 2024-04-24T23:50:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.165+0000 7f42f9f42640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f42f406b9b0 con 0x7f42f40b8060 2024-04-24T23:50:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.165+0000 7f42f3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f415dfd0 0x7f42f415acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.165+0000 7f42f3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f415dfd0 0x7f42f415acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56900/0 (socket says 172.21.15.16:56900) 2024-04-24T23:50:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.165+0000 7f42f3fff640 1 -- 172.21.15.16:0/729089747 learned_addr learned my addr 172.21.15.16:0/729089747 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.166+0000 7f42f3fff640 1 -- 172.21.15.16:0/729089747 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f42f40b8060 msgr2=0x7f42f415a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.166+0000 7f42f3fff640 1 --2- 172.21.15.16:0/729089747 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f42f40b8060 0x7f42f415a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.166+0000 7f42f3fff640 1 -- 172.21.15.16:0/729089747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42f406bc40 msgr2=0x7f42f40cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.166+0000 7f42f3fff640 1 --2- 172.21.15.16:0/729089747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42f406bc40 0x7f42f40cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.166+0000 7f42f3fff640 1 -- 172.21.15.16:0/729089747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42f40cbaa0 con 0x7f42f415dfd0 2024-04-24T23:50:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.166+0000 7f42f3fff640 1 --2- 172.21.15.16:0/729089747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f415dfd0 0x7f42f415acf0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00cef0 tx=0x7f42ec00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.166+0000 7f42dbfff640 1 -- 172.21.15.16:0/729089747 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec01a070 con 0x7f42f415dfd0 2024-04-24T23:50:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.166+0000 7f42dbfff640 1 -- 172.21.15.16:0/729089747 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42ec00d860 con 0x7f42f415dfd0 2024-04-24T23:50:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.166+0000 7f42dbfff640 1 -- 172.21.15.16:0/729089747 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ec00dc50 con 0x7f42f415dfd0 2024-04-24T23:50:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.166+0000 7f42f9f42640 1 -- 172.21.15.16:0/729089747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42f40cbd80 con 0x7f42f415dfd0 2024-04-24T23:50:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.166+0000 7f42f9f42640 1 -- 172.21.15.16:0/729089747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42f40cc340 con 0x7f42f415dfd0 2024-04-24T23:50:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.167+0000 7f42f9f42640 1 -- 172.21.15.16:0/729089747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42f40b4440 con 0x7f42f415dfd0 2024-04-24T23:50:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.167+0000 7f42dbfff640 1 -- 172.21.15.16:0/729089747 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f42ec01d070 con 0x7f42f415dfd0 2024-04-24T23:50:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.168+0000 7f42dbfff640 1 --2- 172.21.15.16:0/729089747 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f42e006be30 0x7f42e006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.168+0000 7f42dbfff640 1 -- 172.21.15.16:0/729089747 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f42ec08bb40 con 0x7f42f415dfd0 2024-04-24T23:50:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.168+0000 7f42f37fe640 1 --2- 172.21.15.16:0/729089747 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f42e006be30 0x7f42e006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.169+0000 7f42f37fe640 1 --2- 172.21.15.16:0/729089747 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f42e006be30 0x7f42e006e2f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f42e4034000 tx=0x7f42e402c000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.172+0000 7f42dbfff640 1 -- 172.21.15.16:0/729089747 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f42f40b4440 con 0x7f42f415dfd0 2024-04-24T23:50:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.353+0000 7f42f9f42640 1 -- 172.21.15.16:0/729089747 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f42f4068f90 con 0x7f42f415dfd0 2024-04-24T23:50:03.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.354+0000 7f42dbfff640 1 -- 172.21.15.16:0/729089747 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f42ec05a1b0 con 0x7f42f415dfd0 2024-04-24T23:50:03.354 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:50:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.356+0000 7f42f9f42640 1 -- 172.21.15.16:0/729089747 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f42e006be30 msgr2=0x7f42e006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.356+0000 7f42f9f42640 1 --2- 172.21.15.16:0/729089747 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f42e006be30 0x7f42e006e2f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f42e4034000 tx=0x7f42e402c000 comp rx=0 tx=0).stop 2024-04-24T23:50:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.356+0000 7f42f9f42640 1 -- 172.21.15.16:0/729089747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f415dfd0 msgr2=0x7f42f415acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.356+0000 7f42f9f42640 1 --2- 172.21.15.16:0/729089747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f415dfd0 0x7f42f415acf0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f42ec00cef0 tx=0x7f42ec00a710 comp rx=0 tx=0).stop 2024-04-24T23:50:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.356+0000 7f42f9f42640 1 -- 172.21.15.16:0/729089747 shutdown_connections 2024-04-24T23:50:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.356+0000 7f42f9f42640 1 --2- 172.21.15.16:0/729089747 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f42e006be30 0x7f42e006e2f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.356+0000 7f42f9f42640 1 --2- 172.21.15.16:0/729089747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f42f415dfd0 0x7f42f415acf0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.356+0000 7f42f9f42640 1 --2- 172.21.15.16:0/729089747 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f42f40b8060 0x7f42f415a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.356+0000 7f42f9f42640 1 --2- 172.21.15.16:0/729089747 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42f406bc40 0x7f42f40cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.356+0000 7f42f9f42640 1 -- 172.21.15.16:0/729089747 >> 172.21.15.16:0/729089747 conn(0x7f42f40d3dc0 msgr2=0x7f42f41603c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.356+0000 7f42f9f42640 1 -- 172.21.15.16:0/729089747 shutdown_connections 2024-04-24T23:50:03.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.356+0000 7f42f9f42640 1 -- 172.21.15.16:0/729089747 wait complete. 2024-04-24T23:50:03.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:50:03.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.470+0000 7fb3d2ce1640 1 Processor -- start 2024-04-24T23:50:03.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.470+0000 7fb3d2ce1640 1 -- start start 2024-04-24T23:50:03.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.471+0000 7fb3d2ce1640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb3cc1478d0 0x7fb3cc0d1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.471+0000 7fb3d2ce1640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3cc0b23a0 0x7fb3cc0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.471+0000 7fb3d2ce1640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3cc0b2a10 con 0x7fb3cc0d2090 2024-04-24T23:50:03.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.471+0000 7fb3d2ce1640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fb3cc0d24f0 con 0x7fb3cc1478d0 2024-04-24T23:50:03.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.471+0000 7fb3d2ce1640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb3cc0b3620 con 0x7fb3cc0b23a0 2024-04-24T23:50:03.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.471+0000 7fb3d0a7d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb3cc1478d0 0x7fb3cc0d1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.471+0000 7fb3d127e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3cc0b23a0 0x7fb3cc0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.471+0000 7fb3d0a7d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb3cc1478d0 0x7fb3cc0d1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:45016/0 (socket says 172.21.15.16:45016) 2024-04-24T23:50:03.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.471+0000 7fb3cbfff640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fb3cc0d2090 0x7fb3cc0b1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:40804/0 (socket says 172.21.15.16:40804) 2024-04-24T23:50:03.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.471+0000 7fb3d0a7d640 1 -- 172.21.15.16:0/2978510095 learned_addr learned my addr 172.21.15.16:0/2978510095 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:03.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.472+0000 7fb3d0a7d640 1 -- 172.21.15.16:0/2978510095 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3cc0b23a0 msgr2=0x7fb3cc0b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:03.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.472+0000 7fb3d0a7d640 1 --2- 172.21.15.16:0/2978510095 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3cc0b23a0 0x7fb3cc0b4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.472+0000 7fb3d0a7d640 1 -- 172.21.15.16:0/2978510095 >> v1:172.21.15.156:6789/0 conn(0x7fb3cc0d2090 legacy=0x7fb3cc0b1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.472+0000 7fb3d0a7d640 1 -- 172.21.15.16:0/2978510095 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3cc0b2cf0 con 0x7fb3cc1478d0 2024-04-24T23:50:03.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.473+0000 7fb3d0a7d640 1 --2- 172.21.15.16:0/2978510095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3cc1478d0 0x7fb3cc0d1b50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009e50 tx=0x7fb3c402a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea6a8d5649239f72 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:03.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.473+0000 7fb3cb7fe640 1 -- 172.21.15.16:0/2978510095 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4036070 con 0x7fb3cc1478d0 2024-04-24T23:50:03.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.473+0000 7fb3cb7fe640 1 -- 172.21.15.16:0/2978510095 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3c4031600 con 0x7fb3cc1478d0 2024-04-24T23:50:03.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.473+0000 7fb3cb7fe640 1 -- 172.21.15.16:0/2978510095 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3c40319d0 con 0x7fb3cc1478d0 2024-04-24T23:50:03.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.473+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/2978510095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3cc1478d0 msgr2=0x7fb3cc0d1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.473+0000 7fb3d2ce1640 1 --2- 172.21.15.16:0/2978510095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3cc1478d0 0x7fb3cc0d1b50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4009e50 tx=0x7fb3c402a710 comp rx=0 tx=0).stop 2024-04-24T23:50:03.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.473+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/2978510095 shutdown_connections 2024-04-24T23:50:03.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.473+0000 7fb3d2ce1640 1 --2- 172.21.15.16:0/2978510095 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3cc0b23a0 0x7fb3cc0b4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.473+0000 7fb3d2ce1640 1 --2- 172.21.15.16:0/2978510095 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3cc1478d0 0x7fb3cc0d1b50 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.473+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/2978510095 >> 172.21.15.16:0/2978510095 conn(0x7fb3cc06b2c0 msgr2=0x7fb3cc072d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:03.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.473+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/2978510095 shutdown_connections 2024-04-24T23:50:03.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.473+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/2978510095 wait complete. 2024-04-24T23:50:03.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.475+0000 7fb3d2ce1640 1 Processor -- start 2024-04-24T23:50:03.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.475+0000 7fb3d2ce1640 1 -- start start 2024-04-24T23:50:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.475+0000 7fb3d2ce1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3cc0b23a0 0x7fb3cc15d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.475+0000 7fb3d2ce1640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3cc0d2090 0x7fb3cc1f3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.475+0000 7fb3d2ce1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3cc1478d0 0x7fb3cc146050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.475+0000 7fb3d2ce1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc0b5350 con 0x7fb3cc1478d0 2024-04-24T23:50:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.475+0000 7fb3d2ce1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb3cc0b32b0 con 0x7fb3cc0b23a0 2024-04-24T23:50:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.475+0000 7fb3d2ce1640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb3cc0b54d0 con 0x7fb3cc0d2090 2024-04-24T23:50:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3d127e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3cc1478d0 0x7fb3cc146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3d127e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3cc1478d0 0x7fb3cc146050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56908/0 (socket says 172.21.15.16:56908) 2024-04-24T23:50:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3d127e640 1 -- 172.21.15.16:0/1560829907 learned_addr learned my addr 172.21.15.16:0/1560829907 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3cbfff640 1 --2- 172.21.15.16:0/1560829907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3cc0d2090 0x7fb3cc1f3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3d0a7d640 1 --2- 172.21.15.16:0/1560829907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3cc0b23a0 0x7fb3cc15d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3d127e640 1 -- 172.21.15.16:0/1560829907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3cc0d2090 msgr2=0x7fb3cc1f3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3d127e640 1 --2- 172.21.15.16:0/1560829907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3cc0d2090 0x7fb3cc1f3e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3d127e640 1 -- 172.21.15.16:0/1560829907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3cc0b23a0 msgr2=0x7fb3cc15d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3d127e640 1 --2- 172.21.15.16:0/1560829907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3cc0b23a0 0x7fb3cc15d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3d127e640 1 -- 172.21.15.16:0/1560829907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3cc1f48a0 con 0x7fb3cc1478d0 2024-04-24T23:50:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3d127e640 1 --2- 172.21.15.16:0/1560829907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3cc1478d0 0x7fb3cc146050 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc00e960 tx=0x7fb3bc00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3c97fa640 1 -- 172.21.15.16:0/1560829907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc017070 con 0x7fb3cc1478d0 2024-04-24T23:50:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3c97fa640 1 -- 172.21.15.16:0/1560829907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3bc00c980 con 0x7fb3cc1478d0 2024-04-24T23:50:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/1560829907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3cc1f4b80 con 0x7fb3cc1478d0 2024-04-24T23:50:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3c97fa640 1 -- 172.21.15.16:0/1560829907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc00cd70 con 0x7fb3cc1478d0 2024-04-24T23:50:03.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.476+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/1560829907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3cc142e90 con 0x7fb3cc1478d0 2024-04-24T23:50:03.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.477+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/1560829907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3cc068f90 con 0x7fb3cc1478d0 2024-04-24T23:50:03.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.478+0000 7fb3c97fa640 1 -- 172.21.15.16:0/1560829907 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb3bc01a070 con 0x7fb3cc1478d0 2024-04-24T23:50:03.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.478+0000 7fb3c97fa640 1 --2- 172.21.15.16:0/1560829907 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb3b806be30 0x7fb3b806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.478+0000 7fb3c97fa640 1 -- 172.21.15.16:0/1560829907 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fb3bc08b1c0 con 0x7fb3cc1478d0 2024-04-24T23:50:03.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.479+0000 7fb3d0a7d640 1 --2- 172.21.15.16:0/1560829907 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb3b806be30 0x7fb3b806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.479+0000 7fb3d0a7d640 1 --2- 172.21.15.16:0/1560829907 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb3b806be30 0x7fb3b806e2f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4033000 tx=0x7fb3c4009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:03.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.483+0000 7fb3c97fa640 1 -- 172.21.15.16:0/1560829907 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb3bc059120 con 0x7fb3cc1478d0 2024-04-24T23:50:03.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.667+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/1560829907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb3cc0b6210 con 0x7fb3cc1478d0 2024-04-24T23:50:03.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.668+0000 7fb3c97fa640 1 -- 172.21.15.16:0/1560829907 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7fb3bc05d420 con 0x7fb3cc1478d0 2024-04-24T23:50:03.668 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:50:03.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.670+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/1560829907 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb3b806be30 msgr2=0x7fb3b806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.670+0000 7fb3d2ce1640 1 --2- 172.21.15.16:0/1560829907 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb3b806be30 0x7fb3b806e2f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4033000 tx=0x7fb3c4009000 comp rx=0 tx=0).stop 2024-04-24T23:50:03.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.670+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/1560829907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3cc1478d0 msgr2=0x7fb3cc146050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.670+0000 7fb3d2ce1640 1 --2- 172.21.15.16:0/1560829907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3cc1478d0 0x7fb3cc146050 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc00e960 tx=0x7fb3bc00ee20 comp rx=0 tx=0).stop 2024-04-24T23:50:03.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.670+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/1560829907 shutdown_connections 2024-04-24T23:50:03.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.670+0000 7fb3d2ce1640 1 --2- 172.21.15.16:0/1560829907 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb3b806be30 0x7fb3b806e2f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.670+0000 7fb3d2ce1640 1 --2- 172.21.15.16:0/1560829907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb3cc1478d0 0x7fb3cc146050 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.670+0000 7fb3d2ce1640 1 --2- 172.21.15.16:0/1560829907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb3cc0d2090 0x7fb3cc1f3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.670+0000 7fb3d2ce1640 1 --2- 172.21.15.16:0/1560829907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb3cc0b23a0 0x7fb3cc15d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.670+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/1560829907 >> 172.21.15.16:0/1560829907 conn(0x7fb3cc06b2c0 msgr2=0x7fb3cc0b6730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:03.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.670+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/1560829907 shutdown_connections 2024-04-24T23:50:03.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.670+0000 7fb3d2ce1640 1 -- 172.21.15.16:0/1560829907 wait complete. 2024-04-24T23:50:03.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:50:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.783+0000 7fe9f11c1640 1 Processor -- start 2024-04-24T23:50:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.783+0000 7fe9f11c1640 1 -- start start 2024-04-24T23:50:03.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9f11c1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe9ec15dfa0 0x7fe9ec0d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9f11c1640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe9ec0ce5f0 0x7fe9ec0cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9f11c1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9ec0d05a0 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9f11c1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fe9ec0b7360 con 0x7fe9ec15dfa0 2024-04-24T23:50:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9f11c1640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fe9ec0cc640 con 0x7fe9ec0ce5f0 2024-04-24T23:50:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9ead9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe9ec15dfa0 0x7fe9ec0d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9eb59e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe9ec0b5f00 0x7fe9ec0b6370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58990/0 (socket says 172.21.15.16:58990) 2024-04-24T23:50:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9eb59e640 1 -- 172.21.15.16:0/510940421 learned_addr learned my addr 172.21.15.16:0/510940421 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9ead9d640 1 -- 172.21.15.16:0/510940421 >> v2:172.21.15.156:3300/0 conn(0x7fe9ec0ce5f0 msgr2=0x7fe9ec0cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:03.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9e3fff640 1 --2- 172.21.15.16:0/510940421 >> v2:172.21.15.156:3300/0 conn(0x7fe9ec0ce5f0 0x7fe9ec0cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9ead9d640 1 --2- 172.21.15.16:0/510940421 >> v2:172.21.15.156:3300/0 conn(0x7fe9ec0ce5f0 0x7fe9ec0cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9ea59c640 1 -- 172.21.15.16:0/510940421 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1763204053 0 0) 0x7fe9ec0d05a0 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9ead9d640 1 -- 172.21.15.16:0/510940421 >> v1:172.21.15.16:6789/0 conn(0x7fe9ec0b5f00 legacy=0x7fe9ec0b6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9ead9d640 1 -- 172.21.15.16:0/510940421 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ec0d0880 con 0x7fe9ec15dfa0 2024-04-24T23:50:03.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.784+0000 7fe9e3fff640 1 --2- 172.21.15.16:0/510940421 >> v2:172.21.15.156:3300/0 conn(0x7fe9ec0ce5f0 0x7fe9ec0cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:03.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.785+0000 7fe9ead9d640 1 --2- 172.21.15.16:0/510940421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec15dfa0 0x7fe9ec0d0fe0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec161a40 tx=0x7fe9dc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d95f6c1caa000ac server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:03.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.785+0000 7fe9ea59c640 1 -- 172.21.15.16:0/510940421 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc037070 con 0x7fe9ec15dfa0 2024-04-24T23:50:03.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.785+0000 7fe9ea59c640 1 -- 172.21.15.16:0/510940421 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9dc032440 con 0x7fe9ec15dfa0 2024-04-24T23:50:03.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.785+0000 7fe9ea59c640 1 -- 172.21.15.16:0/510940421 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc032810 con 0x7fe9ec15dfa0 2024-04-24T23:50:03.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.786+0000 7fe9f11c1640 1 -- 172.21.15.16:0/510940421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec15dfa0 msgr2=0x7fe9ec0d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.786+0000 7fe9f11c1640 1 --2- 172.21.15.16:0/510940421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec15dfa0 0x7fe9ec0d0fe0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec161a40 tx=0x7fe9dc02a1a0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.786+0000 7fe9f11c1640 1 -- 172.21.15.16:0/510940421 shutdown_connections 2024-04-24T23:50:03.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.786+0000 7fe9f11c1640 1 --2- 172.21.15.16:0/510940421 >> v2:172.21.15.156:3300/0 conn(0x7fe9ec0ce5f0 0x7fe9ec0cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.786+0000 7fe9f11c1640 1 --2- 172.21.15.16:0/510940421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec15dfa0 0x7fe9ec0d0fe0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.786+0000 7fe9f11c1640 1 -- 172.21.15.16:0/510940421 >> 172.21.15.16:0/510940421 conn(0x7fe9ec06b4d0 msgr2=0x7fe9ec0b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:03.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.786+0000 7fe9f11c1640 1 -- 172.21.15.16:0/510940421 shutdown_connections 2024-04-24T23:50:03.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.786+0000 7fe9f11c1640 1 -- 172.21.15.16:0/510940421 wait complete. 2024-04-24T23:50:03.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.787+0000 7fe9f11c1640 1 Processor -- start 2024-04-24T23:50:03.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.787+0000 7fe9f11c1640 1 -- start start 2024-04-24T23:50:03.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9f11c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec0b5f00 0x7fe9ec0c9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9f11c1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9ec0ce5f0 0x7fe9ec1558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9f11c1640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9ec15dfa0 0x7fe9ec14a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9f11c1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec15e380 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9f11c1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe9ec0cc2d0 con 0x7fe9ec0ce5f0 2024-04-24T23:50:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9f11c1640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe9ec0d1520 con 0x7fe9ec15dfa0 2024-04-24T23:50:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9ead9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec0b5f00 0x7fe9ec0c9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9ead9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec0b5f00 0x7fe9ec0c9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56912/0 (socket says 172.21.15.16:56912) 2024-04-24T23:50:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9ead9d640 1 -- 172.21.15.16:0/4125230797 learned_addr learned my addr 172.21.15.16:0/4125230797 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9e3fff640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9ec0ce5f0 0x7fe9ec1558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9ead9d640 1 -- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9ec15dfa0 msgr2=0x7fe9ec14a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9eb59e640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9ec15dfa0 0x7fe9ec14a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9ead9d640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9ec15dfa0 0x7fe9ec14a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9ead9d640 1 -- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9ec0ce5f0 msgr2=0x7fe9ec1558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9ead9d640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9ec0ce5f0 0x7fe9ec1558a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9ead9d640 1 -- 172.21.15.16:0/4125230797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9ec14a5d0 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9e3fff640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9ec0ce5f0 0x7fe9ec1558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9eb59e640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9ec15dfa0 0x7fe9ec14a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.788+0000 7fe9ead9d640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec0b5f00 0x7fe9ec0c9850 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009890 tx=0x7fe9dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.789+0000 7fe9e37fe640 1 -- 172.21.15.16:0/4125230797 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc037040 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.789+0000 7fe9e37fe640 1 -- 172.21.15.16:0/4125230797 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9dc042070 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.789+0000 7fe9f11c1640 1 -- 172.21.15.16:0/4125230797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9ec154800 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.789+0000 7fe9e37fe640 1 -- 172.21.15.16:0/4125230797 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc030a50 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.789+0000 7fe9f11c1640 1 -- 172.21.15.16:0/4125230797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9ec154dc0 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.790+0000 7fe9f11c1640 1 -- 172.21.15.16:0/4125230797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe9ec06bea0 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.790+0000 7fe9e37fe640 1 -- 172.21.15.16:0/4125230797 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe9dc030bf0 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.791+0000 7fe9e37fe640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe9cc06be30 0x7fe9cc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.791+0000 7fe9e37fe640 1 -- 172.21.15.16:0/4125230797 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fe9dc0abc80 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.791+0000 7fe9e3fff640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe9cc06be30 0x7fe9cc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:03.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.792+0000 7fe9e3fff640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe9cc06be30 0x7fe9cc06e2f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4006fd0 tx=0x7fe9d4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:03.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.795+0000 7fe9e37fe640 1 -- 172.21.15.16:0/4125230797 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe9ec14a5d0 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.990+0000 7fe9f11c1640 1 -- 172.21.15.16:0/4125230797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fe9ec04f5e0 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.991+0000 7fe9e37fe640 1 -- 172.21.15.16:0/4125230797 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) ==== 80+0+0 (secure 0 0 0) 0x7fe9dc07a2c0 con 0x7fe9ec0b5f00 2024-04-24T23:50:03.992 INFO:teuthology.orchestra.run.smithi016.stderr:listed 0 entries 2024-04-24T23:50:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.993+0000 7fe9f11c1640 1 -- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe9cc06be30 msgr2=0x7fe9cc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.993+0000 7fe9f11c1640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe9cc06be30 0x7fe9cc06e2f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe9d4006fd0 tx=0x7fe9d4008040 comp rx=0 tx=0).stop 2024-04-24T23:50:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.993+0000 7fe9f11c1640 1 -- 172.21.15.16:0/4125230797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec0b5f00 msgr2=0x7fe9ec0c9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.993+0000 7fe9f11c1640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec0b5f00 0x7fe9ec0c9850 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009890 tx=0x7fe9dc0046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.993+0000 7fe9f11c1640 1 -- 172.21.15.16:0/4125230797 shutdown_connections 2024-04-24T23:50:03.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.993+0000 7fe9f11c1640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe9cc06be30 0x7fe9cc06e2f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.993+0000 7fe9f11c1640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe9ec15dfa0 0x7fe9ec14a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.993+0000 7fe9f11c1640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe9ec0ce5f0 0x7fe9ec1558a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.993+0000 7fe9f11c1640 1 --2- 172.21.15.16:0/4125230797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe9ec0b5f00 0x7fe9ec0c9850 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.993+0000 7fe9f11c1640 1 -- 172.21.15.16:0/4125230797 >> 172.21.15.16:0/4125230797 conn(0x7fe9ec06b4d0 msgr2=0x7fe9ec0cdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:03.995 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.993+0000 7fe9f11c1640 1 -- 172.21.15.16:0/4125230797 shutdown_connections 2024-04-24T23:50:03.996 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:03.993+0000 7fe9f11c1640 1 -- 172.21.15.16:0/4125230797 wait complete. 2024-04-24T23:50:04.004 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi156.front.sepia.ceph.com... 2024-04-24T23:50:04.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:04.004 DEBUG:teuthology.orchestra.run.smithi156:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-04-24T23:50:04.035 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:50:04.036 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:50:04.036 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:04.036 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-24T23:50:04.094 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-04-24T23:50:04.095 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:50:04.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.196+0000 7f411305d640 1 Processor -- start 2024-04-24T23:50:04.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.196+0000 7f411305d640 1 -- start start 2024-04-24T23:50:04.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.197+0000 7f411305d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f410c06cb00 0x7f410c15c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.197+0000 7f411305d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f410c0729a0 0x7f410c0ced40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.197+0000 7f411305d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f410c06cee0 con 0x7f410c0ccc70 2024-04-24T23:50:04.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.197+0000 7f411305d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f410c0ce760 con 0x7f410c0729a0 2024-04-24T23:50:04.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.197+0000 7f411305d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f410c15c8b0 con 0x7f410c06cb00 2024-04-24T23:50:04.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.197+0000 7f4110df9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f410c06cb00 0x7f410c15c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.197+0000 7f4110df9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f410c06cb00 0x7f410c15c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:56644/0 (socket says 172.21.15.16:56644) 2024-04-24T23:50:04.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.197+0000 7f4110df9640 1 -- 172.21.15.16:0/2346434518 learned_addr learned my addr 172.21.15.16:0/2346434518 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:04.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.197+0000 7f41115fa640 1 --2- 172.21.15.16:0/2346434518 >> v2:172.21.15.156:3300/0 conn(0x7f410c0729a0 0x7f410c0ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.198+0000 7f4110df9640 1 -- 172.21.15.16:0/2346434518 >> v2:172.21.15.156:3300/0 conn(0x7f410c0729a0 msgr2=0x7f410c0ced40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:04.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.198+0000 7f4110df9640 1 --2- 172.21.15.16:0/2346434518 >> v2:172.21.15.156:3300/0 conn(0x7f410c0729a0 0x7f410c0ced40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.198+0000 7f4110df9640 1 -- 172.21.15.16:0/2346434518 >> v1:172.21.15.156:6789/0 conn(0x7f410c0ccc70 legacy=0x7f410c072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.198+0000 7f4110df9640 1 -- 172.21.15.16:0/2346434518 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f410c15cb90 con 0x7f410c06cb00 2024-04-24T23:50:04.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.198+0000 7f41115fa640 1 --2- 172.21.15.16:0/2346434518 >> v2:172.21.15.156:3300/0 conn(0x7f410c0729a0 0x7f410c0ced40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:50:04.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.198+0000 7f4110df9640 1 --2- 172.21.15.16:0/2346434518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f410c06cb00 0x7f410c15c250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4100009870 tx=0x7f410002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3de21c0897df86c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:04.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.199+0000 7f410b7fe640 1 -- 172.21.15.16:0/2346434518 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4100037070 con 0x7f410c06cb00 2024-04-24T23:50:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.199+0000 7f410b7fe640 1 -- 172.21.15.16:0/2346434518 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4100032440 con 0x7f410c06cb00 2024-04-24T23:50:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.199+0000 7f410b7fe640 1 -- 172.21.15.16:0/2346434518 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4100032810 con 0x7f410c06cb00 2024-04-24T23:50:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.199+0000 7f411305d640 1 -- 172.21.15.16:0/2346434518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f410c06cb00 msgr2=0x7f410c15c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.199+0000 7f411305d640 1 --2- 172.21.15.16:0/2346434518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f410c06cb00 0x7f410c15c250 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4100009870 tx=0x7f410002a270 comp rx=0 tx=0).stop 2024-04-24T23:50:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.199+0000 7f411305d640 1 -- 172.21.15.16:0/2346434518 shutdown_connections 2024-04-24T23:50:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.199+0000 7f411305d640 1 --2- 172.21.15.16:0/2346434518 >> v2:172.21.15.156:3300/0 conn(0x7f410c0729a0 0x7f410c0ced40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.199+0000 7f411305d640 1 --2- 172.21.15.16:0/2346434518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f410c06cb00 0x7f410c15c250 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.199+0000 7f411305d640 1 -- 172.21.15.16:0/2346434518 >> 172.21.15.16:0/2346434518 conn(0x7f410c0d0260 msgr2=0x7f410c0d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:04.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.199+0000 7f411305d640 1 -- 172.21.15.16:0/2346434518 shutdown_connections 2024-04-24T23:50:04.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.199+0000 7f411305d640 1 -- 172.21.15.16:0/2346434518 wait complete. 2024-04-24T23:50:04.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.200+0000 7f411305d640 1 Processor -- start 2024-04-24T23:50:04.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.200+0000 7f411305d640 1 -- start start 2024-04-24T23:50:04.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.201+0000 7f411305d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f410c06cb00 0x7f410c1f4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.201+0000 7f411305d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c0729a0 0x7f410c0aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.201+0000 7f411305d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f410c0ccc70 0x7f410c155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.201+0000 7f411305d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f410c06b0e0 con 0x7f410c0729a0 2024-04-24T23:50:04.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.201+0000 7f411305d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f410c15d150 con 0x7f410c06cb00 2024-04-24T23:50:04.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.201+0000 7f411305d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f410c0cd0f0 con 0x7f410c0ccc70 2024-04-24T23:50:04.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.201+0000 7f410bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c0729a0 0x7f410c0aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.201+0000 7f410bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c0729a0 0x7f410c0aedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56924/0 (socket says 172.21.15.16:56924) 2024-04-24T23:50:04.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.201+0000 7f410bfff640 1 -- 172.21.15.16:0/3540894866 learned_addr learned my addr 172.21.15.16:0/3540894866 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:04.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.201+0000 7f4110df9640 1 --2- 172.21.15.16:0/3540894866 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f410c06cb00 0x7f410c1f4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.201+0000 7f41115fa640 1 --2- 172.21.15.16:0/3540894866 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f410c0ccc70 0x7f410c155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.202+0000 7f410bfff640 1 -- 172.21.15.16:0/3540894866 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f410c0ccc70 msgr2=0x7f410c155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.202+0000 7f410bfff640 1 --2- 172.21.15.16:0/3540894866 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f410c0ccc70 0x7f410c155640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.202+0000 7f410bfff640 1 -- 172.21.15.16:0/3540894866 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f410c06cb00 msgr2=0x7f410c1f4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.202+0000 7f410bfff640 1 --2- 172.21.15.16:0/3540894866 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f410c06cb00 0x7f410c1f4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.202+0000 7f410bfff640 1 -- 172.21.15.16:0/3540894866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f410c0af300 con 0x7f410c0729a0 2024-04-24T23:50:04.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.202+0000 7f410bfff640 1 --2- 172.21.15.16:0/3540894866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c0729a0 0x7f410c0aedc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f40f800c400 tx=0x7f40f800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:04.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.202+0000 7f41097fa640 1 -- 172.21.15.16:0/3540894866 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f8017070 con 0x7f410c0729a0 2024-04-24T23:50:04.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.202+0000 7f41097fa640 1 -- 172.21.15.16:0/3540894866 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40f80052d0 con 0x7f410c0729a0 2024-04-24T23:50:04.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.202+0000 7f411305d640 1 -- 172.21.15.16:0/3540894866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f410c1f4790 con 0x7f410c0729a0 2024-04-24T23:50:04.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.202+0000 7f41097fa640 1 -- 172.21.15.16:0/3540894866 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f8003830 con 0x7f410c0729a0 2024-04-24T23:50:04.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.202+0000 7f411305d640 1 -- 172.21.15.16:0/3540894866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f410c1f5030 con 0x7f410c0729a0 2024-04-24T23:50:04.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.203+0000 7f411305d640 1 -- 172.21.15.16:0/3540894866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f410c04f5e0 con 0x7f410c0729a0 2024-04-24T23:50:04.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.204+0000 7f41097fa640 1 -- 172.21.15.16:0/3540894866 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f40f801a070 con 0x7f410c0729a0 2024-04-24T23:50:04.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.204+0000 7f41097fa640 1 --2- 172.21.15.16:0/3540894866 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f40f006be30 0x7f40f006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.204+0000 7f41097fa640 1 -- 172.21.15.16:0/3540894866 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f40f808a470 con 0x7f410c0729a0 2024-04-24T23:50:04.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.204+0000 7f4110df9640 1 --2- 172.21.15.16:0/3540894866 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f40f006be30 0x7f40f006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.205+0000 7f4110df9640 1 --2- 172.21.15.16:0/3540894866 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f40f006be30 0x7f40f006e2f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4100034000 tx=0x7f410002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:04.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.208+0000 7f41097fa640 1 -- 172.21.15.16:0/3540894866 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f40f80583d0 con 0x7f410c0729a0 2024-04-24T23:50:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.428+0000 7f411305d640 1 -- 172.21.15.16:0/3540894866 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f410c06c2d0 con 0x7f410c0729a0 2024-04-24T23:50:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.429+0000 7f41097fa640 1 -- 172.21.15.16:0/3540894866 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5180 (secure 0 0 0) 0x7f40f805c6d0 con 0x7f410c0729a0 2024-04-24T23:50:04.429 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:04.429 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1623032341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1623032341},{"type":"v1","addr":"172.21.15.156:6836","nonce":1623032341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3530738236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3530738236},{"type":"v1","addr":"172.21.15.16:6835","nonce":3530738236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2153063131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2153063131},{"type":"v1","addr":"172.21.15.16:6834","nonce":2153063131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:48:54.852787+0000","modified":"2024-04-24T23:49:17.914730+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/289127714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":289127714},{"type":"v1","addr":"172.21.15.156:6837","nonce":289127714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-24T23:50:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-04-24T23:50:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.431+0000 7f411305d640 1 -- 172.21.15.16:0/3540894866 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f40f006be30 msgr2=0x7f40f006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.431+0000 7f411305d640 1 --2- 172.21.15.16:0/3540894866 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f40f006be30 0x7f40f006e2f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4100034000 tx=0x7f410002e000 comp rx=0 tx=0).stop 2024-04-24T23:50:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.431+0000 7f411305d640 1 -- 172.21.15.16:0/3540894866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c0729a0 msgr2=0x7f410c0aedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.431+0000 7f411305d640 1 --2- 172.21.15.16:0/3540894866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c0729a0 0x7f410c0aedc0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f40f800c400 tx=0x7f40f800c800 comp rx=0 tx=0).stop 2024-04-24T23:50:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.431+0000 7f411305d640 1 -- 172.21.15.16:0/3540894866 shutdown_connections 2024-04-24T23:50:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.431+0000 7f411305d640 1 --2- 172.21.15.16:0/3540894866 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f40f006be30 0x7f40f006e2f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.431+0000 7f411305d640 1 --2- 172.21.15.16:0/3540894866 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f410c0ccc70 0x7f410c155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.431+0000 7f411305d640 1 --2- 172.21.15.16:0/3540894866 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f410c0729a0 0x7f410c0aedc0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.431+0000 7f411305d640 1 --2- 172.21.15.16:0/3540894866 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f410c06cb00 0x7f410c1f4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.431+0000 7f411305d640 1 -- 172.21.15.16:0/3540894866 >> 172.21.15.16:0/3540894866 conn(0x7f410c0d0260 msgr2=0x7f410c0b0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:04.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.431+0000 7f411305d640 1 -- 172.21.15.16:0/3540894866 shutdown_connections 2024-04-24T23:50:04.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.431+0000 7f411305d640 1 -- 172.21.15.16:0/3540894866 wait complete. 2024-04-24T23:50:04.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:50:04.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.546+0000 7fa2733c1640 1 Processor -- start 2024-04-24T23:50:04.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.546+0000 7fa2733c1640 1 -- start start 2024-04-24T23:50:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.547+0000 7fa2733c1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa26c15dfa0 0x7fa26c0d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.547+0000 7fa2733c1640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa26c0cdae0 0x7fa26c0cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.547+0000 7fa2733c1640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa26c0d05a0 con 0x7fa26c0ce5f0 2024-04-24T23:50:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.547+0000 7fa2733c1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa26c0b7360 con 0x7fa26c15dfa0 2024-04-24T23:50:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.547+0000 7fa2733c1640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fa26c0cc640 con 0x7fa26c0cdae0 2024-04-24T23:50:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.547+0000 7fa27115d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa26c15dfa0 0x7fa26c0d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.547+0000 7fa27095c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa26c0ce5f0 0x7fa26c0b5f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58994/0 (socket says 172.21.15.16:58994) 2024-04-24T23:50:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.547+0000 7fa27095c640 1 -- 172.21.15.16:0/1898689249 learned_addr learned my addr 172.21.15.16:0/1898689249 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:04.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.547+0000 7fa27195e640 1 --2- 172.21.15.16:0/1898689249 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa26c0cdae0 0x7fa26c0cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.547+0000 7fa27115d640 1 -- 172.21.15.16:0/1898689249 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa26c0cdae0 msgr2=0x7fa26c0cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:04.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.547+0000 7fa27115d640 1 --2- 172.21.15.16:0/1898689249 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa26c0cdae0 0x7fa26c0cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.547+0000 7fa27115d640 1 -- 172.21.15.16:0/1898689249 >> v1:172.21.15.16:6789/0 conn(0x7fa26c0ce5f0 legacy=0x7fa26c0b5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.548 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.547+0000 7fa27115d640 1 -- 172.21.15.16:0/1898689249 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26c0d0880 con 0x7fa26c15dfa0 2024-04-24T23:50:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.548+0000 7fa27115d640 1 --2- 172.21.15.16:0/1898689249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa26c15dfa0 0x7fa26c0d0fe0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa2680097a0 tx=0x7fa26802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e9eaaff0b5f9403 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.548+0000 7fa263fff640 1 -- 172.21.15.16:0/1898689249 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa268037070 con 0x7fa26c15dfa0 2024-04-24T23:50:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.548+0000 7fa263fff640 1 -- 172.21.15.16:0/1898689249 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa268032440 con 0x7fa26c15dfa0 2024-04-24T23:50:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.548+0000 7fa263fff640 1 -- 172.21.15.16:0/1898689249 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa268032810 con 0x7fa26c15dfa0 2024-04-24T23:50:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.548+0000 7fa2733c1640 1 -- 172.21.15.16:0/1898689249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa26c15dfa0 msgr2=0x7fa26c0d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.548+0000 7fa2733c1640 1 --2- 172.21.15.16:0/1898689249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa26c15dfa0 0x7fa26c0d0fe0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa2680097a0 tx=0x7fa26802a180 comp rx=0 tx=0).stop 2024-04-24T23:50:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.548+0000 7fa2733c1640 1 -- 172.21.15.16:0/1898689249 shutdown_connections 2024-04-24T23:50:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.549+0000 7fa2733c1640 1 --2- 172.21.15.16:0/1898689249 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa26c0cdae0 0x7fa26c0cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.549+0000 7fa2733c1640 1 --2- 172.21.15.16:0/1898689249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa26c15dfa0 0x7fa26c0d0fe0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.549+0000 7fa2733c1640 1 -- 172.21.15.16:0/1898689249 >> 172.21.15.16:0/1898689249 conn(0x7fa26c06b4d0 msgr2=0x7fa26c0b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.549+0000 7fa2733c1640 1 -- 172.21.15.16:0/1898689249 shutdown_connections 2024-04-24T23:50:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.549+0000 7fa2733c1640 1 -- 172.21.15.16:0/1898689249 wait complete. 2024-04-24T23:50:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.550+0000 7fa2733c1640 1 Processor -- start 2024-04-24T23:50:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.550+0000 7fa2733c1640 1 -- start start 2024-04-24T23:50:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.550+0000 7fa2733c1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa26c0cdae0 0x7fa26c070100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.550+0000 7fa2733c1640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa26c0ce5f0 0x7fa26c0c75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.550+0000 7fa2733c1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa26c15dfa0 0x7fa26c150b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.550+0000 7fa2733c1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa26c15e380 con 0x7fa26c15dfa0 2024-04-24T23:50:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.550+0000 7fa2733c1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa26c0cc2d0 con 0x7fa26c0cdae0 2024-04-24T23:50:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa2733c1640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fa26c0d1520 con 0x7fa26c0ce5f0 2024-04-24T23:50:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa27195e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa26c15dfa0 0x7fa26c150b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa27195e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa26c15dfa0 0x7fa26c150b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56946/0 (socket says 172.21.15.16:56946) 2024-04-24T23:50:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa27195e640 1 -- 172.21.15.16:0/1013772711 learned_addr learned my addr 172.21.15.16:0/1013772711 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa27095c640 1 --2- 172.21.15.16:0/1013772711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa26c0ce5f0 0x7fa26c0c75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa27115d640 1 --2- 172.21.15.16:0/1013772711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa26c0cdae0 0x7fa26c070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa27195e640 1 -- 172.21.15.16:0/1013772711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa26c0ce5f0 msgr2=0x7fa26c0c75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa27195e640 1 --2- 172.21.15.16:0/1013772711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa26c0ce5f0 0x7fa26c0c75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa27195e640 1 -- 172.21.15.16:0/1013772711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa26c0cdae0 msgr2=0x7fa26c070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa27195e640 1 --2- 172.21.15.16:0/1013772711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa26c0cdae0 0x7fa26c070100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa27195e640 1 -- 172.21.15.16:0/1013772711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26c0c5310 con 0x7fa26c15dfa0 2024-04-24T23:50:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa27195e640 1 --2- 172.21.15.16:0/1013772711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa26c15dfa0 0x7fa26c150b20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa264007e30 tx=0x7fa26400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa261ffb640 1 -- 172.21.15.16:0/1013772711 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa264014620 con 0x7fa26c15dfa0 2024-04-24T23:50:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa261ffb640 1 -- 172.21.15.16:0/1013772711 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa264019070 con 0x7fa26c15dfa0 2024-04-24T23:50:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa2733c1640 1 -- 172.21.15.16:0/1013772711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa26c0c7f80 con 0x7fa26c15dfa0 2024-04-24T23:50:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.551+0000 7fa261ffb640 1 -- 172.21.15.16:0/1013772711 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa264014a00 con 0x7fa26c15dfa0 2024-04-24T23:50:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.552+0000 7fa2733c1640 1 -- 172.21.15.16:0/1013772711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa26c0c8260 con 0x7fa26c15dfa0 2024-04-24T23:50:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.553+0000 7fa2733c1640 1 -- 172.21.15.16:0/1013772711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa26c06bea0 con 0x7fa26c15dfa0 2024-04-24T23:50:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.553+0000 7fa261ffb640 1 -- 172.21.15.16:0/1013772711 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa264014ba0 con 0x7fa26c15dfa0 2024-04-24T23:50:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.553+0000 7fa261ffb640 1 --2- 172.21.15.16:0/1013772711 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa25006be30 0x7fa25006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.553+0000 7fa261ffb640 1 -- 172.21.15.16:0/1013772711 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa26408bb80 con 0x7fa26c15dfa0 2024-04-24T23:50:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.554+0000 7fa27115d640 1 --2- 172.21.15.16:0/1013772711 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa25006be30 0x7fa25006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.554+0000 7fa27115d640 1 --2- 172.21.15.16:0/1013772711 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa25006be30 0x7fa25006e2f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa268034000 tx=0x7fa26802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:04.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.558+0000 7fa261ffb640 1 -- 172.21.15.16:0/1013772711 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa26405a130 con 0x7fa26c15dfa0 2024-04-24T23:50:04.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.743+0000 7fa2733c1640 1 -- 172.21.15.16:0/1013772711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa26c04f610 con 0x7fa26c15dfa0 2024-04-24T23:50:04.744 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.744+0000 7fa261ffb640 1 -- 172.21.15.16:0/1013772711 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7fa264059a30 con 0x7fa26c15dfa0 2024-04-24T23:50:04.744 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:04.745 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:50:04.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.746+0000 7fa2733c1640 1 -- 172.21.15.16:0/1013772711 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa25006be30 msgr2=0x7fa25006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.746+0000 7fa2733c1640 1 --2- 172.21.15.16:0/1013772711 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa25006be30 0x7fa25006e2f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa268034000 tx=0x7fa26802e000 comp rx=0 tx=0).stop 2024-04-24T23:50:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.746+0000 7fa2733c1640 1 -- 172.21.15.16:0/1013772711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa26c15dfa0 msgr2=0x7fa26c150b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.746+0000 7fa2733c1640 1 --2- 172.21.15.16:0/1013772711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa26c15dfa0 0x7fa26c150b20 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa264007e30 tx=0x7fa26400e4b0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.746+0000 7fa2733c1640 1 -- 172.21.15.16:0/1013772711 shutdown_connections 2024-04-24T23:50:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.746+0000 7fa2733c1640 1 --2- 172.21.15.16:0/1013772711 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa25006be30 0x7fa25006e2f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.746+0000 7fa2733c1640 1 --2- 172.21.15.16:0/1013772711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa26c15dfa0 0x7fa26c150b20 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.747+0000 7fa2733c1640 1 --2- 172.21.15.16:0/1013772711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa26c0ce5f0 0x7fa26c0c75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.747+0000 7fa2733c1640 1 --2- 172.21.15.16:0/1013772711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa26c0cdae0 0x7fa26c070100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.747+0000 7fa2733c1640 1 -- 172.21.15.16:0/1013772711 >> 172.21.15.16:0/1013772711 conn(0x7fa26c06b4d0 msgr2=0x7fa26c0cad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.747+0000 7fa2733c1640 1 -- 172.21.15.16:0/1013772711 shutdown_connections 2024-04-24T23:50:04.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.747+0000 7fa2733c1640 1 -- 172.21.15.16:0/1013772711 wait complete. 2024-04-24T23:50:04.758 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:48:53.157933+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:48:53.823351+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-24T23:48:57.531302+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-24T23:50:04.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:50:04.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.863+0000 7f8bd7377640 1 Processor -- start 2024-04-24T23:50:04.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.863+0000 7f8bd7377640 1 -- start start 2024-04-24T23:50:04.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.864+0000 7f8bd7377640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8bd015d0a0 0x7f8bd00ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.864+0000 7f8bd7377640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8bd00ceca0 0x7f8bd006ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.864+0000 7f8bd7377640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8bd015f9c0 con 0x7f8bd01455f0 2024-04-24T23:50:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.864+0000 7f8bd7377640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8bd00cf100 con 0x7f8bd015d0a0 2024-04-24T23:50:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.864+0000 7f8bd7377640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8bd01605d0 con 0x7f8bd00ceca0 2024-04-24T23:50:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.864+0000 7f8bd4912640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8bd015d0a0 0x7f8bd00ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.864+0000 7f8bd5113640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8bd01455f0 0x7f8bd006b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59002/0 (socket says 172.21.15.16:59002) 2024-04-24T23:50:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.864+0000 7f8bd5113640 1 -- 172.21.15.16:0/3090866689 learned_addr learned my addr 172.21.15.16:0/3090866689 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.864+0000 7f8bd5914640 1 --2- 172.21.15.16:0/3090866689 >> v2:172.21.15.156:3300/0 conn(0x7f8bd00ceca0 0x7f8bd006ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.864+0000 7f8bd4912640 1 -- 172.21.15.16:0/3090866689 >> v2:172.21.15.156:3300/0 conn(0x7f8bd00ceca0 msgr2=0x7f8bd006ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.864+0000 7f8bd4912640 1 --2- 172.21.15.16:0/3090866689 >> v2:172.21.15.156:3300/0 conn(0x7f8bd00ceca0 0x7f8bd006ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.864+0000 7f8bd4912640 1 -- 172.21.15.16:0/3090866689 >> v1:172.21.15.16:6789/0 conn(0x7f8bd01455f0 legacy=0x7f8bd006b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.864+0000 7f8bd4912640 1 -- 172.21.15.16:0/3090866689 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd015fca0 con 0x7f8bd015d0a0 2024-04-24T23:50:04.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.865+0000 7f8bd4912640 1 --2- 172.21.15.16:0/3090866689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd015d0a0 0x7f8bd00ce760 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0072b30 tx=0x7f8bc002a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56371c151f4e3c1a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:04.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.865+0000 7f8bc7fff640 1 -- 172.21.15.16:0/3090866689 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc0037070 con 0x7f8bd015d0a0 2024-04-24T23:50:04.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.865+0000 7f8bc7fff640 1 -- 172.21.15.16:0/3090866689 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bc0032440 con 0x7f8bd015d0a0 2024-04-24T23:50:04.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.865+0000 7f8bc7fff640 1 -- 172.21.15.16:0/3090866689 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc002ae60 con 0x7f8bd015d0a0 2024-04-24T23:50:04.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.865+0000 7f8bd7377640 1 -- 172.21.15.16:0/3090866689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd015d0a0 msgr2=0x7f8bd00ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.865+0000 7f8bd7377640 1 --2- 172.21.15.16:0/3090866689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd015d0a0 0x7f8bd00ce760 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8bd0072b30 tx=0x7f8bc002a110 comp rx=0 tx=0).stop 2024-04-24T23:50:04.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.865+0000 7f8bd7377640 1 -- 172.21.15.16:0/3090866689 shutdown_connections 2024-04-24T23:50:04.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.865+0000 7f8bd7377640 1 --2- 172.21.15.16:0/3090866689 >> v2:172.21.15.156:3300/0 conn(0x7f8bd00ceca0 0x7f8bd006ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.865+0000 7f8bd7377640 1 --2- 172.21.15.16:0/3090866689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd015d0a0 0x7f8bd00ce760 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.865+0000 7f8bd7377640 1 -- 172.21.15.16:0/3090866689 >> 172.21.15.16:0/3090866689 conn(0x7f8bd00cfc50 msgr2=0x7f8bd00d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:04.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.866+0000 7f8bd7377640 1 -- 172.21.15.16:0/3090866689 shutdown_connections 2024-04-24T23:50:04.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.866+0000 7f8bd7377640 1 -- 172.21.15.16:0/3090866689 wait complete. 2024-04-24T23:50:04.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.867+0000 7f8bd7377640 1 Processor -- start 2024-04-24T23:50:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.867+0000 7f8bd7377640 1 -- start start 2024-04-24T23:50:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.867+0000 7f8bd7377640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd00ceca0 0x7f8bd012ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.867+0000 7f8bd7377640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bd01455f0 0x7f8bd015b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.867+0000 7f8bd7377640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bd015d0a0 0x7f8bd012cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.867+0000 7f8bd7377640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd006b7f0 con 0x7f8bd00ceca0 2024-04-24T23:50:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.867+0000 7f8bd7377640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8bd0160260 con 0x7f8bd015d0a0 2024-04-24T23:50:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.867+0000 7f8bd7377640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8bd01459d0 con 0x7f8bd01455f0 2024-04-24T23:50:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bd5113640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd00ceca0 0x7f8bd012ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bd5113640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd00ceca0 0x7f8bd012ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56974/0 (socket says 172.21.15.16:56974) 2024-04-24T23:50:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bd5113640 1 -- 172.21.15.16:0/2345228279 learned_addr learned my addr 172.21.15.16:0/2345228279 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bd4912640 1 --2- 172.21.15.16:0/2345228279 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bd01455f0 0x7f8bd015b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bd5914640 1 --2- 172.21.15.16:0/2345228279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bd015d0a0 0x7f8bd012cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bd5113640 1 -- 172.21.15.16:0/2345228279 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bd01455f0 msgr2=0x7f8bd015b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bd5113640 1 --2- 172.21.15.16:0/2345228279 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bd01455f0 0x7f8bd015b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bd5113640 1 -- 172.21.15.16:0/2345228279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bd015d0a0 msgr2=0x7f8bd012cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:04.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bd5113640 1 --2- 172.21.15.16:0/2345228279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bd015d0a0 0x7f8bd012cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:04.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bd5113640 1 -- 172.21.15.16:0/2345228279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bd012d960 con 0x7f8bd00ceca0 2024-04-24T23:50:04.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bd5113640 1 --2- 172.21.15.16:0/2345228279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd00ceca0 0x7f8bd012ea00 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800c400 tx=0x7f8bc800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:04.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bc5ffb640 1 -- 172.21.15.16:0/2345228279 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc8017070 con 0x7f8bd00ceca0 2024-04-24T23:50:04.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bc5ffb640 1 -- 172.21.15.16:0/2345228279 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8bc8005ce0 con 0x7f8bd00ceca0 2024-04-24T23:50:04.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bc5ffb640 1 -- 172.21.15.16:0/2345228279 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bc80072f0 con 0x7f8bd00ceca0 2024-04-24T23:50:04.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.868+0000 7f8bd7377640 1 -- 172.21.15.16:0/2345228279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bd012dc40 con 0x7f8bd00ceca0 2024-04-24T23:50:04.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.869+0000 7f8bd7377640 1 -- 172.21.15.16:0/2345228279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bd012e200 con 0x7f8bd00ceca0 2024-04-24T23:50:04.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.870+0000 7f8bd7377640 1 -- 172.21.15.16:0/2345228279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bd0068f90 con 0x7f8bd00ceca0 2024-04-24T23:50:04.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.870+0000 7f8bc5ffb640 1 -- 172.21.15.16:0/2345228279 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8bc801a070 con 0x7f8bd00ceca0 2024-04-24T23:50:04.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.870+0000 7f8bc5ffb640 1 --2- 172.21.15.16:0/2345228279 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8bb406be60 0x7f8bb406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:04.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.870+0000 7f8bc5ffb640 1 -- 172.21.15.16:0/2345228279 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8bc808a5f0 con 0x7f8bd00ceca0 2024-04-24T23:50:04.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.871+0000 7f8bd4912640 1 --2- 172.21.15.16:0/2345228279 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8bb406be60 0x7f8bb406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:04.871 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.871+0000 7f8bd4912640 1 --2- 172.21.15.16:0/2345228279 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8bb406be60 0x7f8bb406e320 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0034000 tx=0x7f8bc002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:04.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:04.875+0000 7f8bc5ffb640 1 -- 172.21.15.16:0/2345228279 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8bc8058550 con 0x7f8bd00ceca0 2024-04-24T23:50:05.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.056+0000 7f8bd7377640 1 -- 172.21.15.16:0/2345228279 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8bd00b3150 con 0x7f8bd00ceca0 2024-04-24T23:50:05.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.056+0000 7f8bc5ffb640 1 -- 172.21.15.16:0/2345228279 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) ==== 93+0+10 (secure 0 0 0) 0x7f8bc805c850 con 0x7f8bd00ceca0 2024-04-24T23:50:05.056 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:50:05.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.058+0000 7f8bd7377640 1 -- 172.21.15.16:0/2345228279 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8bb406be60 msgr2=0x7f8bb406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.058+0000 7f8bd7377640 1 --2- 172.21.15.16:0/2345228279 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8bb406be60 0x7f8bb406e320 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8bc0034000 tx=0x7f8bc002e000 comp rx=0 tx=0).stop 2024-04-24T23:50:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.058+0000 7f8bd7377640 1 -- 172.21.15.16:0/2345228279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd00ceca0 msgr2=0x7f8bd012ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.058+0000 7f8bd7377640 1 --2- 172.21.15.16:0/2345228279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd00ceca0 0x7f8bd012ea00 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f8bc800c400 tx=0x7f8bc800c8c0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.059+0000 7f8bd7377640 1 -- 172.21.15.16:0/2345228279 shutdown_connections 2024-04-24T23:50:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.059+0000 7f8bd7377640 1 --2- 172.21.15.16:0/2345228279 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8bb406be60 0x7f8bb406e320 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.059+0000 7f8bd7377640 1 --2- 172.21.15.16:0/2345228279 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8bd015d0a0 0x7f8bd012cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.059+0000 7f8bd7377640 1 --2- 172.21.15.16:0/2345228279 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8bd01455f0 0x7f8bd015b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.059+0000 7f8bd7377640 1 --2- 172.21.15.16:0/2345228279 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd00ceca0 0x7f8bd012ea00 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.059+0000 7f8bd7377640 1 -- 172.21.15.16:0/2345228279 >> 172.21.15.16:0/2345228279 conn(0x7f8bd00cfc50 msgr2=0x7f8bd015d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.059+0000 7f8bd7377640 1 -- 172.21.15.16:0/2345228279 shutdown_connections 2024-04-24T23:50:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.059+0000 7f8bd7377640 1 -- 172.21.15.16:0/2345228279 wait complete. 2024-04-24T23:50:05.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:50:05.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.174+0000 7fa5d5ec1640 1 Processor -- start 2024-04-24T23:50:05.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.174+0000 7fa5d5ec1640 1 -- start start 2024-04-24T23:50:05.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.175+0000 7fa5d5ec1640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa5d01444c0 0x7fa5d0071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.175+0000 7fa5d5ec1640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa5d0071fd0 0x7fa5d00ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.175+0000 7fa5d5ec1640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5d00cf0a0 con 0x7fa5d00ce870 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.175+0000 7fa5d5ec1640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fa5d0072430 con 0x7fa5d01444c0 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.175+0000 7fa5d5ec1640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa5d006bbf0 con 0x7fa5d0071fd0 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.175+0000 7fa5cf7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa5d01444c0 0x7fa5d0071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.175+0000 7fa5cf7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa5d01444c0 0x7fa5d0071a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56990/0 (socket says 172.21.15.16:56990) 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.175+0000 7fa5cf7fe640 1 -- 172.21.15.16:0/3911621002 learned_addr learned my addr 172.21.15.16:0/3911621002 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.175+0000 7fa5cf7fe640 1 -- 172.21.15.16:0/3911621002 >> v2:172.21.15.156:3300/0 conn(0x7fa5d0071fd0 msgr2=0x7fa5d00ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.175+0000 7fa5cf7fe640 1 --2- 172.21.15.16:0/3911621002 >> v2:172.21.15.156:3300/0 conn(0x7fa5d0071fd0 0x7fa5d00ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.175+0000 7fa5cf7fe640 1 -- 172.21.15.16:0/3911621002 >> v1:172.21.15.156:6789/0 conn(0x7fa5d00ce870 legacy=0x7fa5d00b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.175+0000 7fa5cf7fe640 1 -- 172.21.15.16:0/3911621002 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d006be90 con 0x7fa5d01444c0 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.176+0000 7fa5cf7fe640 1 --2- 172.21.15.16:0/3911621002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d01444c0 0x7fa5d0071a90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa5d006ec30 tx=0x7fa5c002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ad795e7f443dee server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.176+0000 7fa5ce7fc640 1 -- 172.21.15.16:0/3911621002 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0037070 con 0x7fa5d01444c0 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.176+0000 7fa5ce7fc640 1 -- 172.21.15.16:0/3911621002 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c0032440 con 0x7fa5d01444c0 2024-04-24T23:50:05.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.176+0000 7fa5ce7fc640 1 -- 172.21.15.16:0/3911621002 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0032810 con 0x7fa5d01444c0 2024-04-24T23:50:05.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.176+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/3911621002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d01444c0 msgr2=0x7fa5d0071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.176+0000 7fa5d5ec1640 1 --2- 172.21.15.16:0/3911621002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d01444c0 0x7fa5d0071a90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa5d006ec30 tx=0x7fa5c002a230 comp rx=0 tx=0).stop 2024-04-24T23:50:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.177+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/3911621002 shutdown_connections 2024-04-24T23:50:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.177+0000 7fa5d5ec1640 1 --2- 172.21.15.16:0/3911621002 >> v2:172.21.15.156:3300/0 conn(0x7fa5d0071fd0 0x7fa5d00ce330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.177+0000 7fa5d5ec1640 1 --2- 172.21.15.16:0/3911621002 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d01444c0 0x7fa5d0071a90 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.177+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/3911621002 >> 172.21.15.16:0/3911621002 conn(0x7fa5d00cfc50 msgr2=0x7fa5d00d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.177+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/3911621002 shutdown_connections 2024-04-24T23:50:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.177+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/3911621002 wait complete. 2024-04-24T23:50:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.178+0000 7fa5d5ec1640 1 Processor -- start 2024-04-24T23:50:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.178+0000 7fa5d5ec1640 1 -- start start 2024-04-24T23:50:05.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5d5ec1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0071fd0 0x7fa5d014acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5d5ec1640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa5d00ce870 0x7fa5d01f1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5d5ec1640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5d01444c0 0x7fa5d013ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5d5ec1640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d006c730 con 0x7fa5d0071fd0 2024-04-24T23:50:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5d5ec1640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa5d006c1d0 con 0x7fa5d01444c0 2024-04-24T23:50:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5d5ec1640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fa5d00cecc0 con 0x7fa5d00ce870 2024-04-24T23:50:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5cf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0071fd0 0x7fa5d014acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5cf7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0071fd0 0x7fa5d014acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56994/0 (socket says 172.21.15.16:56994) 2024-04-24T23:50:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5cf7fe640 1 -- 172.21.15.16:0/2693487262 learned_addr learned my addr 172.21.15.16:0/2693487262 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5ceffd640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa5d00ce870 0x7fa5d01f1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5cffff640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5d01444c0 0x7fa5d013ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5cf7fe640 1 -- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa5d00ce870 msgr2=0x7fa5d01f1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5cf7fe640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa5d00ce870 0x7fa5d01f1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5cf7fe640 1 -- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5d01444c0 msgr2=0x7fa5d013ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5cf7fe640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5d01444c0 0x7fa5d013ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5cf7fe640 1 -- 172.21.15.16:0/2693487262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5d013bf70 con 0x7fa5d0071fd0 2024-04-24T23:50:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5ceffd640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa5d00ce870 0x7fa5d01f1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5cffff640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5d01444c0 0x7fa5d013ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5cf7fe640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0071fd0 0x7fa5d014acc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa5c002a270 tx=0x7fa5c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5b7fff640 1 -- 172.21.15.16:0/2693487262 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0041070 con 0x7fa5d0071fd0 2024-04-24T23:50:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5b7fff640 1 -- 172.21.15.16:0/2693487262 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5c003c870 con 0x7fa5d0071fd0 2024-04-24T23:50:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/2693487262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5d014b350 con 0x7fa5d0071fd0 2024-04-24T23:50:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.179+0000 7fa5b7fff640 1 -- 172.21.15.16:0/2693487262 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5c0037040 con 0x7fa5d0071fd0 2024-04-24T23:50:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.180+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/2693487262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5d014b910 con 0x7fa5d0071fd0 2024-04-24T23:50:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.180+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/2693487262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5d0068f90 con 0x7fa5d0071fd0 2024-04-24T23:50:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.181+0000 7fa5b7fff640 1 -- 172.21.15.16:0/2693487262 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa5c0044070 con 0x7fa5d0071fd0 2024-04-24T23:50:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.181+0000 7fa5b7fff640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa5bc06be30 0x7fa5bc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.182+0000 7fa5b7fff640 1 -- 172.21.15.16:0/2693487262 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fa5c00abd40 con 0x7fa5d0071fd0 2024-04-24T23:50:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.182+0000 7fa5ceffd640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa5bc06be30 0x7fa5bc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.182+0000 7fa5ceffd640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa5bc06be30 0x7fa5bc06e2f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa5d01f5670 tx=0x7fa5c4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.186+0000 7fa5b7fff640 1 -- 172.21.15.16:0/2693487262 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa5d013bf70 con 0x7fa5d0071fd0 2024-04-24T23:50:05.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.373+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/2693487262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fa5d015b000 con 0x7fa5d0071fd0 2024-04-24T23:50:05.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.374+0000 7fa5b7fff640 1 -- 172.21.15.16:0/2693487262 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) ==== 104+0+11 (secure 0 0 0) 0x7fa5c007a380 con 0x7fa5d0071fd0 2024-04-24T23:50:05.374 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:50:05.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.376+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa5bc06be30 msgr2=0x7fa5bc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.376+0000 7fa5d5ec1640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa5bc06be30 0x7fa5bc06e2f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa5d01f5670 tx=0x7fa5c4008040 comp rx=0 tx=0).stop 2024-04-24T23:50:05.377 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.376+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/2693487262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0071fd0 msgr2=0x7fa5d014acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.376+0000 7fa5d5ec1640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0071fd0 0x7fa5d014acc0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa5c002a270 tx=0x7fa5c00046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.376+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/2693487262 shutdown_connections 2024-04-24T23:50:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.376+0000 7fa5d5ec1640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa5bc06be30 0x7fa5bc06e2f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.376+0000 7fa5d5ec1640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa5d01444c0 0x7fa5d013ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.376+0000 7fa5d5ec1640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa5d00ce870 0x7fa5d01f1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.376+0000 7fa5d5ec1640 1 --2- 172.21.15.16:0/2693487262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa5d0071fd0 0x7fa5d014acc0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.376+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/2693487262 >> 172.21.15.16:0/2693487262 conn(0x7fa5d00cfc50 msgr2=0x7fa5d00aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.376+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/2693487262 shutdown_connections 2024-04-24T23:50:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.376+0000 7fa5d5ec1640 1 -- 172.21.15.16:0/2693487262 wait complete. 2024-04-24T23:50:05.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:50:05.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.490+0000 7f09f61b7640 1 Processor -- start 2024-04-24T23:50:05.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.491+0000 7f09f61b7640 1 -- start start 2024-04-24T23:50:05.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.491+0000 7f09f61b7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09f015dfd0 0x7f09f006b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.491+0000 7f09f61b7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09f006bc40 0x7f09f006c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.491+0000 7f09f61b7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09f00d24d0 con 0x7f09f00cefc0 2024-04-24T23:50:05.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.491+0000 7f09f61b7640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f09f006c5e0 con 0x7f09f015dfd0 2024-04-24T23:50:05.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.491+0000 7f09f61b7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f09f00d27b0 con 0x7f09f006bc40 2024-04-24T23:50:05.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.491+0000 7f09ef7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09f015dfd0 0x7f09f006b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.492+0000 7f09ef7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09f015dfd0 0x7f09f006b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57004/0 (socket says 172.21.15.16:57004) 2024-04-24T23:50:05.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.492+0000 7f09ef7fe640 1 -- 172.21.15.16:0/2740645358 learned_addr learned my addr 172.21.15.16:0/2740645358 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:05.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.492+0000 7f09eeffd640 1 --2- 172.21.15.16:0/2740645358 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09f006bc40 0x7f09f006c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.499 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.492+0000 7f09ef7fe640 1 -- 172.21.15.16:0/2740645358 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09f006bc40 msgr2=0x7f09f006c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:05.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.492+0000 7f09ef7fe640 1 --2- 172.21.15.16:0/2740645358 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09f006bc40 0x7f09f006c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.492+0000 7f09ef7fe640 1 -- 172.21.15.16:0/2740645358 >> v1:172.21.15.156:6789/0 conn(0x7f09f00cefc0 legacy=0x7f09f00cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:05.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.492+0000 7f09ef7fe640 1 -- 172.21.15.16:0/2740645358 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f00d2a90 con 0x7f09f015dfd0 2024-04-24T23:50:05.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.492+0000 7f09eeffd640 1 --2- 172.21.15.16:0/2740645358 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09f006bc40 0x7f09f006c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:05.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.493+0000 7f09ef7fe640 1 --2- 172.21.15.16:0/2740645358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f015dfd0 0x7f09f006b2f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f09e0009870 tx=0x7f09e002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e0e0f4481e2306f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:05.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.493+0000 7f09ee7fc640 1 -- 172.21.15.16:0/2740645358 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0037070 con 0x7f09f015dfd0 2024-04-24T23:50:05.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.493+0000 7f09ee7fc640 1 -- 172.21.15.16:0/2740645358 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e0032440 con 0x7f09f015dfd0 2024-04-24T23:50:05.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.493+0000 7f09ee7fc640 1 -- 172.21.15.16:0/2740645358 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e0032810 con 0x7f09f015dfd0 2024-04-24T23:50:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.493+0000 7f09f61b7640 1 -- 172.21.15.16:0/2740645358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f015dfd0 msgr2=0x7f09f006b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.493+0000 7f09f61b7640 1 --2- 172.21.15.16:0/2740645358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f015dfd0 0x7f09f006b2f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f09e0009870 tx=0x7f09e002a180 comp rx=0 tx=0).stop 2024-04-24T23:50:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.493+0000 7f09f61b7640 1 -- 172.21.15.16:0/2740645358 shutdown_connections 2024-04-24T23:50:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.493+0000 7f09f61b7640 1 --2- 172.21.15.16:0/2740645358 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09f006bc40 0x7f09f006c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.493+0000 7f09f61b7640 1 --2- 172.21.15.16:0/2740645358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f015dfd0 0x7f09f006b2f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.493+0000 7f09f61b7640 1 -- 172.21.15.16:0/2740645358 >> 172.21.15.16:0/2740645358 conn(0x7f09f00d3dc0 msgr2=0x7f09f00d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.493+0000 7f09f61b7640 1 -- 172.21.15.16:0/2740645358 shutdown_connections 2024-04-24T23:50:05.501 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.493+0000 7f09f61b7640 1 -- 172.21.15.16:0/2740645358 wait complete. 2024-04-24T23:50:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.494+0000 7f09f61b7640 1 Processor -- start 2024-04-24T23:50:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.495+0000 7f09f61b7640 1 -- start start 2024-04-24T23:50:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.495+0000 7f09f61b7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09f006bc40 0x7f09f01f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.495+0000 7f09f61b7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09f00cefc0 0x7f09f01603d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.495+0000 7f09f61b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f015dfd0 0x7f09f0160910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.495+0000 7f09f61b7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f09f00cead0 con 0x7f09f015dfd0 2024-04-24T23:50:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.495+0000 7f09f61b7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f09f00d3050 con 0x7f09f006bc40 2024-04-24T23:50:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.495+0000 7f09f61b7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f09f006b830 con 0x7f09f00cefc0 2024-04-24T23:50:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.495+0000 7f09effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f015dfd0 0x7f09f0160910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09effff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f015dfd0 0x7f09f0160910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57008/0 (socket says 172.21.15.16:57008) 2024-04-24T23:50:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09effff640 1 -- 172.21.15.16:0/2668681251 learned_addr learned my addr 172.21.15.16:0/2668681251 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:05.502 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09ef7fe640 1 --2- 172.21.15.16:0/2668681251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09f006bc40 0x7f09f01f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09eeffd640 1 --2- 172.21.15.16:0/2668681251 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09f00cefc0 0x7f09f01603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09effff640 1 -- 172.21.15.16:0/2668681251 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09f00cefc0 msgr2=0x7f09f01603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09effff640 1 --2- 172.21.15.16:0/2668681251 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09f00cefc0 0x7f09f01603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09effff640 1 -- 172.21.15.16:0/2668681251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09f006bc40 msgr2=0x7f09f01f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09effff640 1 --2- 172.21.15.16:0/2668681251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09f006bc40 0x7f09f01f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09effff640 1 -- 172.21.15.16:0/2668681251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f00cd780 con 0x7f09f015dfd0 2024-04-24T23:50:05.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09effff640 1 --2- 172.21.15.16:0/2668681251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f015dfd0 0x7f09f0160910 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f09e800c780 tx=0x7f09e800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:05.503 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09d7fff640 1 -- 172.21.15.16:0/2668681251 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e8004b60 con 0x7f09f015dfd0 2024-04-24T23:50:05.505 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09d7fff640 1 -- 172.21.15.16:0/2668681251 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09e8004d00 con 0x7f09f015dfd0 2024-04-24T23:50:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09d7fff640 1 -- 172.21.15.16:0/2668681251 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09e800aa90 con 0x7f09f015dfd0 2024-04-24T23:50:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09f61b7640 1 -- 172.21.15.16:0/2668681251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f00cda60 con 0x7f09f015dfd0 2024-04-24T23:50:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.496+0000 7f09f61b7640 1 -- 172.21.15.16:0/2668681251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f00ce020 con 0x7f09f015dfd0 2024-04-24T23:50:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.497+0000 7f09f61b7640 1 -- 172.21.15.16:0/2668681251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f0068f90 con 0x7f09f015dfd0 2024-04-24T23:50:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.498+0000 7f09d7fff640 1 -- 172.21.15.16:0/2668681251 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f09e800ac30 con 0x7f09f015dfd0 2024-04-24T23:50:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.498+0000 7f09d7fff640 1 --2- 172.21.15.16:0/2668681251 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f09dc06be30 0x7f09dc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.498+0000 7f09d7fff640 1 -- 172.21.15.16:0/2668681251 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f09e808b3a0 con 0x7f09f015dfd0 2024-04-24T23:50:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.498+0000 7f09ef7fe640 1 --2- 172.21.15.16:0/2668681251 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f09dc06be30 0x7f09dc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.506 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.499+0000 7f09ef7fe640 1 --2- 172.21.15.16:0/2668681251 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f09dc06be30 0x7f09dc06e2f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f09e0034000 tx=0x7f09e002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:05.507 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.502+0000 7f09d7fff640 1 -- 172.21.15.16:0/2668681251 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f09e8020070 con 0x7f09f015dfd0 2024-04-24T23:50:05.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.685+0000 7f09f61b7640 1 -- 172.21.15.16:0/2668681251 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f09f015cf60 con 0x7f09f015dfd0 2024-04-24T23:50:05.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.685+0000 7f09d7fff640 1 -- 172.21.15.16:0/2668681251 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) ==== 100+0+11 (secure 0 0 0) 0x7f09e8059390 con 0x7f09f015dfd0 2024-04-24T23:50:05.685 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:50:05.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.687+0000 7f09f61b7640 1 -- 172.21.15.16:0/2668681251 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f09dc06be30 msgr2=0x7f09dc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.687+0000 7f09f61b7640 1 --2- 172.21.15.16:0/2668681251 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f09dc06be30 0x7f09dc06e2f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f09e0034000 tx=0x7f09e002e000 comp rx=0 tx=0).stop 2024-04-24T23:50:05.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.687+0000 7f09f61b7640 1 -- 172.21.15.16:0/2668681251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f015dfd0 msgr2=0x7f09f0160910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.687+0000 7f09f61b7640 1 --2- 172.21.15.16:0/2668681251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f015dfd0 0x7f09f0160910 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f09e800c780 tx=0x7f09e800cc40 comp rx=0 tx=0).stop 2024-04-24T23:50:05.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.687+0000 7f09f61b7640 1 -- 172.21.15.16:0/2668681251 shutdown_connections 2024-04-24T23:50:05.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.687+0000 7f09f61b7640 1 --2- 172.21.15.16:0/2668681251 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f09dc06be30 0x7f09dc06e2f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.687+0000 7f09f61b7640 1 --2- 172.21.15.16:0/2668681251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09f015dfd0 0x7f09f0160910 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.687+0000 7f09f61b7640 1 --2- 172.21.15.16:0/2668681251 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f09f00cefc0 0x7f09f01603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.687+0000 7f09f61b7640 1 --2- 172.21.15.16:0/2668681251 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f09f006bc40 0x7f09f01f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.687+0000 7f09f61b7640 1 -- 172.21.15.16:0/2668681251 >> 172.21.15.16:0/2668681251 conn(0x7f09f00d3dc0 msgr2=0x7f09f00b8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:05.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.687+0000 7f09f61b7640 1 -- 172.21.15.16:0/2668681251 shutdown_connections 2024-04-24T23:50:05.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.687+0000 7f09f61b7640 1 -- 172.21.15.16:0/2668681251 wait complete. 2024-04-24T23:50:05.698 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-24T23:50:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.801+0000 7f8e75363640 1 Processor -- start 2024-04-24T23:50:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.801+0000 7f8e75363640 1 -- start start 2024-04-24T23:50:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.802+0000 7f8e75363640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8e700b85d0 0x7f8e70168e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.802+0000 7f8e75363640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e7015e6d0 con 0x7f8e700d26e0 2024-04-24T23:50:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.802+0000 7f8e75363640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e7006c230 con 0x7f8e7015e2f0 2024-04-24T23:50:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.802+0000 7f8e75363640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8e700d2ca0 con 0x7f8e700b85d0 2024-04-24T23:50:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.802+0000 7f8e6f7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8e700b85d0 0x7f8e70168e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.802+0000 7f8e6e7fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8e700d26e0 0x7f8e7006bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59018/0 (socket says 172.21.15.16:59018) 2024-04-24T23:50:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.802+0000 7f8e6e7fc640 1 -- 172.21.15.16:0/3113736249 learned_addr learned my addr 172.21.15.16:0/3113736249 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.802+0000 7f8e6f7fe640 1 -- 172.21.15.16:0/3113736249 >> v1:172.21.15.156:6789/0 conn(0x7f8e7015e2f0 legacy=0x7f8e7006b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.802+0000 7f8e6f7fe640 1 -- 172.21.15.16:0/3113736249 >> v1:172.21.15.16:6789/0 conn(0x7f8e700d26e0 legacy=0x7f8e7006bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.802+0000 7f8e6dffb640 1 -- 172.21.15.16:0/3113736249 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1439484420 0 0) 0x7f8e7015e6d0 con 0x7f8e700d26e0 2024-04-24T23:50:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.803+0000 7f8e6f7fe640 1 -- 172.21.15.16:0/3113736249 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e700d2f80 con 0x7f8e700b85d0 2024-04-24T23:50:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.803+0000 7f8e6f7fe640 1 --2- 172.21.15.16:0/3113736249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e700b85d0 0x7f8e70168e00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009d90 tx=0x7f8e6002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518aa6a5855c3694 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.803+0000 7f8e6dffb640 1 -- 172.21.15.16:0/3113736249 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60007740 con 0x7f8e700b85d0 2024-04-24T23:50:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.803+0000 7f8e6dffb640 1 -- 172.21.15.16:0/3113736249 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e600078e0 con 0x7f8e700b85d0 2024-04-24T23:50:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.803+0000 7f8e6dffb640 1 -- 172.21.15.16:0/3113736249 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e60007cb0 con 0x7f8e700b85d0 2024-04-24T23:50:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.804+0000 7f8e75363640 1 -- 172.21.15.16:0/3113736249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e700b85d0 msgr2=0x7f8e70168e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.804+0000 7f8e75363640 1 --2- 172.21.15.16:0/3113736249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e700b85d0 0x7f8e70168e00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009d90 tx=0x7f8e6002a6a0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.804+0000 7f8e75363640 1 -- 172.21.15.16:0/3113736249 shutdown_connections 2024-04-24T23:50:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.804+0000 7f8e75363640 1 --2- 172.21.15.16:0/3113736249 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e700b85d0 0x7f8e70168e00 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.804+0000 7f8e75363640 1 -- 172.21.15.16:0/3113736249 >> 172.21.15.16:0/3113736249 conn(0x7f8e700d4850 msgr2=0x7f8e700d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.804+0000 7f8e75363640 1 -- 172.21.15.16:0/3113736249 shutdown_connections 2024-04-24T23:50:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.804+0000 7f8e75363640 1 -- 172.21.15.16:0/3113736249 wait complete. 2024-04-24T23:50:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.805+0000 7f8e75363640 1 Processor -- start 2024-04-24T23:50:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.805+0000 7f8e75363640 1 -- start start 2024-04-24T23:50:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e75363640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e700b85d0 0x7f8e70157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e75363640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e700d26e0 0x7f8e700c9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e75363640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e7015e2f0 0x7f8e700c9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e75363640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8e700b70a0 con 0x7f8e700d26e0 2024-04-24T23:50:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e75363640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8e700d3540 con 0x7f8e700b85d0 2024-04-24T23:50:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e75363640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8e700d0080 con 0x7f8e7015e2f0 2024-04-24T23:50:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e6e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e700d26e0 0x7f8e700c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e6e7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e700d26e0 0x7f8e700c9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57028/0 (socket says 172.21.15.16:57028) 2024-04-24T23:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e6e7fc640 1 -- 172.21.15.16:0/3459932518 learned_addr learned my addr 172.21.15.16:0/3459932518 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e6effd640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e700b85d0 0x7f8e70157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e6f7fe640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e7015e2f0 0x7f8e700c9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e6e7fc640 1 -- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e7015e2f0 msgr2=0x7f8e700c9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e6e7fc640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e7015e2f0 0x7f8e700c9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e6e7fc640 1 -- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e700b85d0 msgr2=0x7f8e70157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e6e7fc640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e700b85d0 0x7f8e70157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e6e7fc640 1 -- 172.21.15.16:0/3459932518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e701594f0 con 0x7f8e700d26e0 2024-04-24T23:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.806+0000 7f8e6effd640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e700b85d0 0x7f8e70157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.807+0000 7f8e6e7fc640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e700d26e0 0x7f8e700c9880 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800c4d0 tx=0x7f8e5800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.807+0000 7f8e677fe640 1 -- 172.21.15.16:0/3459932518 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e58018070 con 0x7f8e700d26e0 2024-04-24T23:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.807+0000 7f8e677fe640 1 -- 172.21.15.16:0/3459932518 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e58007500 con 0x7f8e700d26e0 2024-04-24T23:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.807+0000 7f8e75363640 1 -- 172.21.15.16:0/3459932518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e701597d0 con 0x7f8e700d26e0 2024-04-24T23:50:05.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.807+0000 7f8e677fe640 1 -- 172.21.15.16:0/3459932518 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e58004a20 con 0x7f8e700d26e0 2024-04-24T23:50:05.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.807+0000 7f8e75363640 1 -- 172.21.15.16:0/3459932518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e70159d90 con 0x7f8e700d26e0 2024-04-24T23:50:05.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.808+0000 7f8e75363640 1 -- 172.21.15.16:0/3459932518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e70068f90 con 0x7f8e700d26e0 2024-04-24T23:50:05.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.808+0000 7f8e677fe640 1 -- 172.21.15.16:0/3459932518 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8e5801b070 con 0x7f8e700d26e0 2024-04-24T23:50:05.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.809+0000 7f8e677fe640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e5006be30 0x7f8e5006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:05.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.809+0000 7f8e677fe640 1 -- 172.21.15.16:0/3459932518 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f8e5808a530 con 0x7f8e700d26e0 2024-04-24T23:50:05.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.809+0000 7f8e6effd640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e5006be30 0x7f8e5006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:05.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.809+0000 7f8e6effd640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e5006be30 0x7f8e5006e2f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8e68006fd0 tx=0x7f8e68008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:05.813 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.813+0000 7f8e677fe640 1 -- 172.21.15.16:0/3459932518 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8e58058ae0 con 0x7f8e700d26e0 2024-04-24T23:50:05.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.998+0000 7f8e75363640 1 -- 172.21.15.16:0/3459932518 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8e700ce860 con 0x7f8e700d26e0 2024-04-24T23:50:05.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:05.999+0000 7f8e677fe640 1 -- 172.21.15.16:0/3459932518 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) ==== 103+0+11 (secure 0 0 0) 0x7f8e580583e0 con 0x7f8e700d26e0 2024-04-24T23:50:05.999 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:50:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.001+0000 7f8e75363640 1 -- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e5006be30 msgr2=0x7f8e5006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.001+0000 7f8e75363640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e5006be30 0x7f8e5006e2f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8e68006fd0 tx=0x7f8e68008040 comp rx=0 tx=0).stop 2024-04-24T23:50:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.001+0000 7f8e75363640 1 -- 172.21.15.16:0/3459932518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e700d26e0 msgr2=0x7f8e700c9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.001+0000 7f8e75363640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e700d26e0 0x7f8e700c9880 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800c4d0 tx=0x7f8e5800c990 comp rx=0 tx=0).stop 2024-04-24T23:50:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.001+0000 7f8e75363640 1 -- 172.21.15.16:0/3459932518 shutdown_connections 2024-04-24T23:50:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.001+0000 7f8e75363640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e5006be30 0x7f8e5006e2f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.001+0000 7f8e75363640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8e7015e2f0 0x7f8e700c9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.001+0000 7f8e75363640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8e700d26e0 0x7f8e700c9880 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.001+0000 7f8e75363640 1 --2- 172.21.15.16:0/3459932518 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8e700b85d0 0x7f8e70157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.001+0000 7f8e75363640 1 -- 172.21.15.16:0/3459932518 >> 172.21.15.16:0/3459932518 conn(0x7f8e700d4850 msgr2=0x7f8e700cc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.001+0000 7f8e75363640 1 -- 172.21.15.16:0/3459932518 shutdown_connections 2024-04-24T23:50:06.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.001+0000 7f8e75363640 1 -- 172.21.15.16:0/3459932518 wait complete. 2024-04-24T23:50:06.012 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:50:06.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.113+0000 7f237975a640 1 Processor -- start 2024-04-24T23:50:06.113 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.113+0000 7f237975a640 1 -- start start 2024-04-24T23:50:06.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.114+0000 7f237975a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f23740cea70 0x7f23740cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.114+0000 7f237975a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23740d05a0 con 0x7f237415dfa0 2024-04-24T23:50:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.114+0000 7f237975a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23740d0880 con 0x7f23740ce5f0 2024-04-24T23:50:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.114+0000 7f237975a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f23740b7360 con 0x7f23740cea70 2024-04-24T23:50:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.114+0000 7f23737fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f23740cea70 0x7f23740cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.114+0000 7f2372ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f237415dfa0 0x7f23740d0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59034/0 (socket says 172.21.15.16:59034) 2024-04-24T23:50:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.114+0000 7f2372ffd640 1 -- 172.21.15.16:0/3951763176 learned_addr learned my addr 172.21.15.16:0/3951763176 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.114+0000 7f2371ffb640 1 -- 172.21.15.16:0/3951763176 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4200901674 0 0) 0x7f23740d05a0 con 0x7f237415dfa0 2024-04-24T23:50:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.114+0000 7f2371ffb640 1 -- 172.21.15.16:0/3951763176 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2360003520 con 0x7f237415dfa0 2024-04-24T23:50:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.114+0000 7f23737fe640 1 -- 172.21.15.16:0/3951763176 >> v1:172.21.15.156:6789/0 conn(0x7f23740ce5f0 legacy=0x7f23740ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.114+0000 7f23737fe640 1 -- 172.21.15.16:0/3951763176 >> v1:172.21.15.16:6789/0 conn(0x7f237415dfa0 legacy=0x7f23740d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.114+0000 7f23737fe640 1 -- 172.21.15.16:0/3951763176 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23740d0b60 con 0x7f23740cea70 2024-04-24T23:50:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.115+0000 7f23737fe640 1 --2- 172.21.15.16:0/3951763176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23740cea70 0x7f23740cd1e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2364009d90 tx=0x7f236402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fac773428db174a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:06.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.115+0000 7f2371ffb640 1 -- 172.21.15.16:0/3951763176 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2364007740 con 0x7f23740cea70 2024-04-24T23:50:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.115+0000 7f2371ffb640 1 -- 172.21.15.16:0/3951763176 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23640078e0 con 0x7f23740cea70 2024-04-24T23:50:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.115+0000 7f2371ffb640 1 -- 172.21.15.16:0/3951763176 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2364007c90 con 0x7f23740cea70 2024-04-24T23:50:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.116+0000 7f237975a640 1 -- 172.21.15.16:0/3951763176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23740cea70 msgr2=0x7f23740cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.116+0000 7f237975a640 1 --2- 172.21.15.16:0/3951763176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23740cea70 0x7f23740cd1e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2364009d90 tx=0x7f236402a6f0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.116+0000 7f237975a640 1 -- 172.21.15.16:0/3951763176 shutdown_connections 2024-04-24T23:50:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.116+0000 7f237975a640 1 --2- 172.21.15.16:0/3951763176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23740cea70 0x7f23740cd1e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.116+0000 7f237975a640 1 -- 172.21.15.16:0/3951763176 >> 172.21.15.16:0/3951763176 conn(0x7f237406b4d0 msgr2=0x7f23740b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.116+0000 7f237975a640 1 -- 172.21.15.16:0/3951763176 shutdown_connections 2024-04-24T23:50:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.116+0000 7f237975a640 1 -- 172.21.15.16:0/3951763176 wait complete. 2024-04-24T23:50:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.117+0000 7f237975a640 1 Processor -- start 2024-04-24T23:50:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.117+0000 7f237975a640 1 -- start start 2024-04-24T23:50:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f237975a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23740ce5f0 0x7f23741e8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f237975a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23740cea70 0x7f23741e77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f237975a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f237415dfa0 0x7f23741e7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f237975a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f23740ceee0 con 0x7f23740ce5f0 2024-04-24T23:50:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f237975a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f23740cc640 con 0x7f23740cea70 2024-04-24T23:50:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f237975a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f237415e380 con 0x7f237415dfa0 2024-04-24T23:50:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f2372ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23740ce5f0 0x7f23741e8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f2372ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23740ce5f0 0x7f23741e8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57042/0 (socket says 172.21.15.16:57042) 2024-04-24T23:50:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f2372ffd640 1 -- 172.21.15.16:0/1699632404 learned_addr learned my addr 172.21.15.16:0/1699632404 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f2372ffd640 1 -- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f237415dfa0 msgr2=0x7f23741e7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f23737fe640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f237415dfa0 0x7f23741e7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f23727fc640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23740cea70 0x7f23741e77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f2372ffd640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f237415dfa0 0x7f23741e7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f2372ffd640 1 -- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23740cea70 msgr2=0x7f23741e77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f2372ffd640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23740cea70 0x7f23741e77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f2372ffd640 1 -- 172.21.15.16:0/1699632404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f237406df50 con 0x7f23740ce5f0 2024-04-24T23:50:06.124 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f23727fc640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23740cea70 0x7f23741e77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f2372ffd640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23740ce5f0 0x7f23741e8a60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f236c00c410 tx=0x7f236c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f236b7fe640 1 -- 172.21.15.16:0/1699632404 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236c017070 con 0x7f23740ce5f0 2024-04-24T23:50:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f236b7fe640 1 -- 172.21.15.16:0/1699632404 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f236c005ce0 con 0x7f23740ce5f0 2024-04-24T23:50:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f237975a640 1 -- 172.21.15.16:0/1699632404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23741e8fa0 con 0x7f23740ce5f0 2024-04-24T23:50:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.118+0000 7f236b7fe640 1 -- 172.21.15.16:0/1699632404 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236c0072f0 con 0x7f23740ce5f0 2024-04-24T23:50:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.119+0000 7f237975a640 1 -- 172.21.15.16:0/1699632404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23740cb3d0 con 0x7f23740ce5f0 2024-04-24T23:50:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.119+0000 7f237975a640 1 -- 172.21.15.16:0/1699632404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23741606f0 con 0x7f23740ce5f0 2024-04-24T23:50:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.120+0000 7f236b7fe640 1 -- 172.21.15.16:0/1699632404 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f236c01a070 con 0x7f23740ce5f0 2024-04-24T23:50:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.120+0000 7f236b7fe640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f235406be30 0x7f235406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.120+0000 7f236b7fe640 1 -- 172.21.15.16:0/1699632404 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f236c08a6a0 con 0x7f23740ce5f0 2024-04-24T23:50:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.120+0000 7f23727fc640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f235406be30 0x7f235406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.125 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.121+0000 7f23727fc640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f235406be30 0x7f235406e2f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f23740b3a90 tx=0x7f235c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:06.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.124+0000 7f236b7fe640 1 -- 172.21.15.16:0/1699632404 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f23741606f0 con 0x7f23740ce5f0 2024-04-24T23:50:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.351+0000 7f237975a640 1 -- 172.21.15.16:0/1699632404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f237406bea0 con 0x7f23740ce5f0 2024-04-24T23:50:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.352+0000 7f236b7fe640 1 -- 172.21.15.16:0/1699632404 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5180 (secure 0 0 0) 0x7f236c058d10 con 0x7f23740ce5f0 2024-04-24T23:50:06.352 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:06.352 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1623032341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1623032341},{"type":"v1","addr":"172.21.15.156:6836","nonce":1623032341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3530738236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3530738236},{"type":"v1","addr":"172.21.15.16:6835","nonce":3530738236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2153063131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2153063131},{"type":"v1","addr":"172.21.15.16:6834","nonce":2153063131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:48:54.852787+0000","modified":"2024-04-24T23:49:17.914730+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/289127714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":289127714},{"type":"v1","addr":"172.21.15.156:6837","nonce":289127714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-24T23:50:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-04-24T23:50:06.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.354+0000 7f237975a640 1 -- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f235406be30 msgr2=0x7f235406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.354+0000 7f237975a640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f235406be30 0x7f235406e2f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f23740b3a90 tx=0x7f235c009040 comp rx=0 tx=0).stop 2024-04-24T23:50:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.354+0000 7f237975a640 1 -- 172.21.15.16:0/1699632404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23740ce5f0 msgr2=0x7f23741e8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.354+0000 7f237975a640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23740ce5f0 0x7f23741e8a60 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f236c00c410 tx=0x7f236c00c8d0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.354+0000 7f237975a640 1 -- 172.21.15.16:0/1699632404 shutdown_connections 2024-04-24T23:50:06.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.354+0000 7f237975a640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f235406be30 0x7f235406e2f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.354+0000 7f237975a640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f237415dfa0 0x7f23741e7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.354+0000 7f237975a640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23740cea70 0x7f23741e77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.354+0000 7f237975a640 1 --2- 172.21.15.16:0/1699632404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f23740ce5f0 0x7f23741e8a60 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.354+0000 7f237975a640 1 -- 172.21.15.16:0/1699632404 >> 172.21.15.16:0/1699632404 conn(0x7f237406b4d0 msgr2=0x7f2374160930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.354+0000 7f237975a640 1 -- 172.21.15.16:0/1699632404 shutdown_connections 2024-04-24T23:50:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.354+0000 7f237975a640 1 -- 172.21.15.16:0/1699632404 wait complete. 2024-04-24T23:50:06.365 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:50:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.467+0000 7f253534f640 1 Processor -- start 2024-04-24T23:50:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.467+0000 7f253534f640 1 -- start start 2024-04-24T23:50:06.471 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.468+0000 7f253534f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f25300cd900 0x7f25300caee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.468+0000 7f253534f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f25300cb420 0x7f25300b5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.468+0000 7f253534f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25300d10b0 con 0x7f253015e070 2024-04-24T23:50:06.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.468+0000 7f253534f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f253015e450 con 0x7f25300cd900 2024-04-24T23:50:06.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.468+0000 7f253534f640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f25300cb850 con 0x7f25300cb420 2024-04-24T23:50:06.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.468+0000 7f252e7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f25300cd900 0x7f25300caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.468+0000 7f252e7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f25300cd900 0x7f25300caee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57044/0 (socket says 172.21.15.16:57044) 2024-04-24T23:50:06.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.468+0000 7f252e7fc640 1 -- 172.21.15.16:0/1973024643 learned_addr learned my addr 172.21.15.16:0/1973024643 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:06.472 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.468+0000 7f252f7fe640 1 --2- 172.21.15.16:0/1973024643 >> v2:172.21.15.156:3300/0 conn(0x7f25300cb420 0x7f25300b5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.469+0000 7f252e7fc640 1 -- 172.21.15.16:0/1973024643 >> v2:172.21.15.156:3300/0 conn(0x7f25300cb420 msgr2=0x7f25300b5890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.469+0000 7f252e7fc640 1 --2- 172.21.15.16:0/1973024643 >> v2:172.21.15.156:3300/0 conn(0x7f25300cb420 0x7f25300b5890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.469+0000 7f252e7fc640 1 -- 172.21.15.16:0/1973024643 >> v1:172.21.15.156:6789/0 conn(0x7f253015e070 legacy=0x7f25300d09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.469+0000 7f252e7fc640 1 -- 172.21.15.16:0/1973024643 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25300cc860 con 0x7f25300cd900 2024-04-24T23:50:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.469+0000 7f252e7fc640 1 --2- 172.21.15.16:0/1973024643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25300cd900 0x7f25300caee0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2530068bf0 tx=0x7f251802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45810b9ca03fbea1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.469+0000 7f252dffb640 1 -- 172.21.15.16:0/1973024643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2518037070 con 0x7f25300cd900 2024-04-24T23:50:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.470+0000 7f252dffb640 1 -- 172.21.15.16:0/1973024643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251802ae50 con 0x7f25300cd900 2024-04-24T23:50:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.470+0000 7f252dffb640 1 -- 172.21.15.16:0/1973024643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2518032860 con 0x7f25300cd900 2024-04-24T23:50:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.470+0000 7f253534f640 1 -- 172.21.15.16:0/1973024643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25300cd900 msgr2=0x7f25300caee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.470+0000 7f253534f640 1 --2- 172.21.15.16:0/1973024643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25300cd900 0x7f25300caee0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f2530068bf0 tx=0x7f251802a300 comp rx=0 tx=0).stop 2024-04-24T23:50:06.473 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.470+0000 7f253534f640 1 -- 172.21.15.16:0/1973024643 shutdown_connections 2024-04-24T23:50:06.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.470+0000 7f253534f640 1 --2- 172.21.15.16:0/1973024643 >> v2:172.21.15.156:3300/0 conn(0x7f25300cb420 0x7f25300b5890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.470+0000 7f253534f640 1 --2- 172.21.15.16:0/1973024643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f25300cd900 0x7f25300caee0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.470+0000 7f253534f640 1 -- 172.21.15.16:0/1973024643 >> 172.21.15.16:0/1973024643 conn(0x7f25300d1bb0 msgr2=0x7f25300d1fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:06.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.470+0000 7f253534f640 1 -- 172.21.15.16:0/1973024643 shutdown_connections 2024-04-24T23:50:06.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.470+0000 7f253534f640 1 -- 172.21.15.16:0/1973024643 wait complete. 2024-04-24T23:50:06.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.471+0000 7f253534f640 1 Processor -- start 2024-04-24T23:50:06.475 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.471+0000 7f253534f640 1 -- start start 2024-04-24T23:50:06.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.472+0000 7f253534f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25300cb420 0x7f25301517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.472+0000 7f253534f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f25300cd900 0x7f2530154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.472+0000 7f253534f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f253015e070 0x7f25300b2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.472+0000 7f253534f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f25300cd100 con 0x7f253015e070 2024-04-24T23:50:06.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.472+0000 7f253534f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f25300cce20 con 0x7f25300cb420 2024-04-24T23:50:06.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.472+0000 7f253534f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f25300cd3e0 con 0x7f25300cd900 2024-04-24T23:50:06.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.472+0000 7f252f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f253015e070 0x7f25300b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.472+0000 7f252f7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f253015e070 0x7f25300b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57060/0 (socket says 172.21.15.16:57060) 2024-04-24T23:50:06.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.472+0000 7f252f7fe640 1 -- 172.21.15.16:0/2578350477 learned_addr learned my addr 172.21.15.16:0/2578350477 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:06.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.472+0000 7f252effd640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25300cb420 0x7f25301517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.472+0000 7f252f7fe640 1 -- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f25300cd900 msgr2=0x7f2530154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.472+0000 7f252e7fc640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f25300cd900 0x7f2530154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.473+0000 7f252f7fe640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f25300cd900 0x7f2530154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.473+0000 7f252f7fe640 1 -- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25300cb420 msgr2=0x7f25301517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.473+0000 7f252f7fe640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25300cb420 0x7f25301517d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.473+0000 7f252f7fe640 1 -- 172.21.15.16:0/2578350477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2530151d10 con 0x7f253015e070 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.473+0000 7f252effd640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25300cb420 0x7f25301517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.473+0000 7f252e7fc640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f25300cd900 0x7f2530154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.473+0000 7f252f7fe640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f253015e070 0x7f25300b2910 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f252000efc0 tx=0x7f252000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.473+0000 7f25277fe640 1 -- 172.21.15.16:0/2578350477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520018020 con 0x7f253015e070 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.473+0000 7f25277fe640 1 -- 172.21.15.16:0/2578350477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2520014480 con 0x7f253015e070 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.473+0000 7f253534f640 1 -- 172.21.15.16:0/2578350477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25301f4290 con 0x7f253015e070 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.473+0000 7f25277fe640 1 -- 172.21.15.16:0/2578350477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520014870 con 0x7f253015e070 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.473+0000 7f253534f640 1 -- 172.21.15.16:0/2578350477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25301f4850 con 0x7f253015e070 2024-04-24T23:50:06.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.474+0000 7f253534f640 1 -- 172.21.15.16:0/2578350477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f253006bfc0 con 0x7f253015e070 2024-04-24T23:50:06.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.474+0000 7f25277fe640 1 -- 172.21.15.16:0/2578350477 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2520005d10 con 0x7f253015e070 2024-04-24T23:50:06.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.475+0000 7f25277fe640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f251006be60 0x7f251006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.475+0000 7f25277fe640 1 -- 172.21.15.16:0/2578350477 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f252008bd30 con 0x7f253015e070 2024-04-24T23:50:06.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.475+0000 7f252effd640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f251006be60 0x7f251006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.476+0000 7f252effd640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f251006be60 0x7f251006e320 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f253015a2a0 tx=0x7f2528009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:06.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.479+0000 7f25277fe640 1 -- 172.21.15.16:0/2578350477 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2520022070 con 0x7f253015e070 2024-04-24T23:50:06.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.666+0000 7f253534f640 1 -- 172.21.15.16:0/2578350477 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f253004f5e0 con 0x7f253015e070 2024-04-24T23:50:06.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.667+0000 7f25277fe640 1 -- 172.21.15.16:0/2578350477 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7f2520060020 con 0x7f253015e070 2024-04-24T23:50:06.668 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:06.668 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:50:06.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.669+0000 7f253534f640 1 -- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f251006be60 msgr2=0x7f251006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.669+0000 7f253534f640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f251006be60 0x7f251006e320 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f253015a2a0 tx=0x7f2528009040 comp rx=0 tx=0).stop 2024-04-24T23:50:06.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.670+0000 7f253534f640 1 -- 172.21.15.16:0/2578350477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f253015e070 msgr2=0x7f25300b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.670+0000 7f253534f640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f253015e070 0x7f25300b2910 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f252000efc0 tx=0x7f252000c490 comp rx=0 tx=0).stop 2024-04-24T23:50:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.670+0000 7f253534f640 1 -- 172.21.15.16:0/2578350477 shutdown_connections 2024-04-24T23:50:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.670+0000 7f253534f640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f251006be60 0x7f251006e320 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.670+0000 7f253534f640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f253015e070 0x7f25300b2910 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.670+0000 7f253534f640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f25300cd900 0x7f2530154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.670+0000 7f253534f640 1 --2- 172.21.15.16:0/2578350477 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25300cb420 0x7f25301517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.670+0000 7f253534f640 1 -- 172.21.15.16:0/2578350477 >> 172.21.15.16:0/2578350477 conn(0x7f25300d1bb0 msgr2=0x7f25300b1760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.670+0000 7f253534f640 1 -- 172.21.15.16:0/2578350477 shutdown_connections 2024-04-24T23:50:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.670+0000 7f253534f640 1 -- 172.21.15.16:0/2578350477 wait complete. 2024-04-24T23:50:06.681 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T23:50:06.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:50:06.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.783+0000 7fda89415640 1 Processor -- start 2024-04-24T23:50:06.784 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.783+0000 7fda89415640 1 -- start start 2024-04-24T23:50:06.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.784+0000 7fda89415640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda840b2bb0 0x7fda840cf980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.784+0000 7fda89415640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fda840d1fa0 0x7fda840d2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.784+0000 7fda89415640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda840b3390 con 0x7fda8415e020 2024-04-24T23:50:06.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.784+0000 7fda89415640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fda8406bfc0 con 0x7fda840d1fa0 2024-04-24T23:50:06.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.784+0000 7fda89415640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fda8406c2a0 con 0x7fda840b2bb0 2024-04-24T23:50:06.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.785+0000 7fda827fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda840b2bb0 0x7fda840cf980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.785+0000 7fda837fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fda840d1fa0 0x7fda840d2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.785+0000 7fda827fc640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda840b2bb0 0x7fda840cf980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:49782/0 (socket says 172.21.15.16:49782) 2024-04-24T23:50:06.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.785+0000 7fda837fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fda840d1fa0 0x7fda840d2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:41098/0 (socket says 172.21.15.16:41098) 2024-04-24T23:50:06.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.785+0000 7fda82ffd640 1 --1- >> v1:172.21.15.156:6789/0 conn(0x7fda8415e020 0x7fda840cf270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:51630/0 (socket says 172.21.15.16:51630) 2024-04-24T23:50:06.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.785+0000 7fda827fc640 1 -- 172.21.15.16:0/165908248 learned_addr learned my addr 172.21.15.16:0/165908248 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:06.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.785+0000 7fda827fc640 1 -- 172.21.15.16:0/165908248 >> v2:172.21.15.156:3300/0 conn(0x7fda840d1fa0 msgr2=0x7fda840d2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:06.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.785+0000 7fda827fc640 1 --2- 172.21.15.16:0/165908248 >> v2:172.21.15.156:3300/0 conn(0x7fda840d1fa0 0x7fda840d2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.785+0000 7fda827fc640 1 -- 172.21.15.16:0/165908248 >> v1:172.21.15.156:6789/0 conn(0x7fda8415e020 legacy=0x7fda840cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.785+0000 7fda827fc640 1 -- 172.21.15.16:0/165908248 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda8406c580 con 0x7fda840b2bb0 2024-04-24T23:50:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.785+0000 7fda837fe640 1 --2- 172.21.15.16:0/165908248 >> v2:172.21.15.156:3300/0 conn(0x7fda840d1fa0 0x7fda840d2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:50:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.786+0000 7fda827fc640 1 --2- 172.21.15.16:0/165908248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda840b2bb0 0x7fda840cf980 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009870 tx=0x7fda6c02a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d0b40d7676fc6b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.786+0000 7fda81ffb640 1 -- 172.21.15.16:0/165908248 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c037070 con 0x7fda840b2bb0 2024-04-24T23:50:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.786+0000 7fda81ffb640 1 -- 172.21.15.16:0/165908248 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda6c02ad60 con 0x7fda840b2bb0 2024-04-24T23:50:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.786+0000 7fda81ffb640 1 -- 172.21.15.16:0/165908248 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c032660 con 0x7fda840b2bb0 2024-04-24T23:50:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.787+0000 7fda89415640 1 -- 172.21.15.16:0/165908248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda840b2bb0 msgr2=0x7fda840cf980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.787+0000 7fda89415640 1 --2- 172.21.15.16:0/165908248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda840b2bb0 0x7fda840cf980 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fda6c009870 tx=0x7fda6c02a0d0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.787+0000 7fda89415640 1 -- 172.21.15.16:0/165908248 shutdown_connections 2024-04-24T23:50:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.787+0000 7fda89415640 1 --2- 172.21.15.16:0/165908248 >> v2:172.21.15.156:3300/0 conn(0x7fda840d1fa0 0x7fda840d2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.787+0000 7fda89415640 1 --2- 172.21.15.16:0/165908248 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda840b2bb0 0x7fda840cf980 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.787+0000 7fda89415640 1 -- 172.21.15.16:0/165908248 >> 172.21.15.16:0/165908248 conn(0x7fda840d3e10 msgr2=0x7fda840d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:06.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.787+0000 7fda89415640 1 -- 172.21.15.16:0/165908248 shutdown_connections 2024-04-24T23:50:06.792 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.787+0000 7fda89415640 1 -- 172.21.15.16:0/165908248 wait complete. 2024-04-24T23:50:06.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.788+0000 7fda89415640 1 Processor -- start 2024-04-24T23:50:06.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.788+0000 7fda89415640 1 -- start start 2024-04-24T23:50:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda89415640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda840b2bb0 0x7fda841f4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda89415640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda840d1fa0 0x7fda84155100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda89415640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda8415e020 0x7fda840cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda89415640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fda8415e400 con 0x7fda840d1fa0 2024-04-24T23:50:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda89415640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fda840d1a80 con 0x7fda8415e020 2024-04-24T23:50:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda89415640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fda840cfec0 con 0x7fda840b2bb0 2024-04-24T23:50:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda827fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda840d1fa0 0x7fda84155100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda827fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda840d1fa0 0x7fda84155100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50502/0 (socket says 172.21.15.16:50502) 2024-04-24T23:50:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda827fc640 1 -- 172.21.15.16:0/825504697 learned_addr learned my addr 172.21.15.16:0/825504697 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda82ffd640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda840b2bb0 0x7fda841f4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda837fe640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda8415e020 0x7fda840cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda827fc640 1 -- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda840b2bb0 msgr2=0x7fda841f4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda827fc640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda840b2bb0 0x7fda841f4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda827fc640 1 -- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda8415e020 msgr2=0x7fda840cc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda827fc640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda8415e020 0x7fda840cc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda827fc640 1 -- 172.21.15.16:0/825504697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda840cce40 con 0x7fda840d1fa0 2024-04-24T23:50:06.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda837fe640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda8415e020 0x7fda840cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.789+0000 7fda827fc640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda840d1fa0 0x7fda84155100 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fda6c000c00 tx=0x7fda6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.790+0000 7fda7b7fe640 1 -- 172.21.15.16:0/825504697 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c037040 con 0x7fda840d1fa0 2024-04-24T23:50:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.790+0000 7fda7b7fe640 1 -- 172.21.15.16:0/825504697 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda6c004030 con 0x7fda840d1fa0 2024-04-24T23:50:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.790+0000 7fda89415640 1 -- 172.21.15.16:0/825504697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda841f4180 con 0x7fda840d1fa0 2024-04-24T23:50:06.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.790+0000 7fda7b7fe640 1 -- 172.21.15.16:0/825504697 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6c004420 con 0x7fda840d1fa0 2024-04-24T23:50:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.790+0000 7fda89415640 1 -- 172.21.15.16:0/825504697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda840cd120 con 0x7fda840d1fa0 2024-04-24T23:50:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.791+0000 7fda89415640 1 -- 172.21.15.16:0/825504697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda8406dfe0 con 0x7fda840d1fa0 2024-04-24T23:50:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.791+0000 7fda7b7fe640 1 -- 172.21.15.16:0/825504697 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fda6c003610 con 0x7fda840d1fa0 2024-04-24T23:50:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.792+0000 7fda7b7fe640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fda6406be30 0x7fda6406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.792+0000 7fda7b7fe640 1 -- 172.21.15.16:0/825504697 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fda6c0abca0 con 0x7fda840d1fa0 2024-04-24T23:50:06.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.792+0000 7fda82ffd640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fda6406be30 0x7fda6406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:06.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.792+0000 7fda82ffd640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fda6406be30 0x7fda6406e2f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fda8415c740 tx=0x7fda7c007ca0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:06.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:06.796+0000 7fda7b7fe640 1 -- 172.21.15.16:0/825504697 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fda6c07a1c0 con 0x7fda840d1fa0 2024-04-24T23:50:07.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.015+0000 7fda89415640 1 -- 172.21.15.16:0/825504697 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda840ce960 con 0x7fda840d1fa0 2024-04-24T23:50:07.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.015+0000 7fda7b7fe640 1 -- 172.21.15.16:0/825504697 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5180 (secure 0 0 0) 0x7fda6c079ac0 con 0x7fda840d1fa0 2024-04-24T23:50:07.015 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:07.015 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1623032341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1623032341},{"type":"v1","addr":"172.21.15.156:6836","nonce":1623032341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3530738236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3530738236},{"type":"v1","addr":"172.21.15.16:6835","nonce":3530738236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2153063131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2153063131},{"type":"v1","addr":"172.21.15.16:6834","nonce":2153063131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:48:54.852787+0000","modified":"2024-04-24T23:49:17.914730+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/289127714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":289127714},{"type":"v1","addr":"172.21.15.156:6837","nonce":289127714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-24T23:50:07.015 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-04-24T23:50:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.017+0000 7fda89415640 1 -- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fda6406be30 msgr2=0x7fda6406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.017+0000 7fda89415640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fda6406be30 0x7fda6406e2f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fda8415c740 tx=0x7fda7c007ca0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.017+0000 7fda89415640 1 -- 172.21.15.16:0/825504697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda840d1fa0 msgr2=0x7fda84155100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.017+0000 7fda89415640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda840d1fa0 0x7fda84155100 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fda6c000c00 tx=0x7fda6c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.017+0000 7fda89415640 1 -- 172.21.15.16:0/825504697 shutdown_connections 2024-04-24T23:50:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.017+0000 7fda89415640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fda6406be30 0x7fda6406e2f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.017+0000 7fda89415640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fda8415e020 0x7fda840cc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.017+0000 7fda89415640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda840d1fa0 0x7fda84155100 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.017+0000 7fda89415640 1 --2- 172.21.15.16:0/825504697 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fda840b2bb0 0x7fda841f4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.017+0000 7fda89415640 1 -- 172.21.15.16:0/825504697 >> 172.21.15.16:0/825504697 conn(0x7fda840d3e10 msgr2=0x7fda8406ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:07.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.017+0000 7fda89415640 1 -- 172.21.15.16:0/825504697 shutdown_connections 2024-04-24T23:50:07.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.017+0000 7fda89415640 1 -- 172.21.15.16:0/825504697 wait complete. 2024-04-24T23:50:07.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:50:07.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.128+0000 7fcfa9431640 1 Processor -- start 2024-04-24T23:50:07.128 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.128+0000 7fcfa9431640 1 -- start start 2024-04-24T23:50:07.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.129+0000 7fcfa9431640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcfa415e020 0x7fcfa40cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.129+0000 7fcfa9431640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fcfa40d1fa0 0x7fcfa40d23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.129+0000 7fcfa9431640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcfa40b3390 con 0x7fcfa40b2bb0 2024-04-24T23:50:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.129+0000 7fcfa9431640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fcfa406bfc0 con 0x7fcfa415e020 2024-04-24T23:50:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.129+0000 7fcfa9431640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fcfa406c2a0 con 0x7fcfa40d1fa0 2024-04-24T23:50:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.129+0000 7fcfa27fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcfa40b2bb0 0x7fcfa40cf7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58994/0 (socket says 172.21.15.16:58994) 2024-04-24T23:50:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.129+0000 7fcfa2ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcfa415e020 0x7fcfa40cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.129+0000 7fcfa27fc640 1 -- 172.21.15.16:0/1141996248 learned_addr learned my addr 172.21.15.16:0/1141996248 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.130+0000 7fcfa37fe640 1 --2- 172.21.15.16:0/1141996248 >> v2:172.21.15.156:3300/0 conn(0x7fcfa40d1fa0 0x7fcfa40d23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.130+0000 7fcfa2ffd640 1 -- 172.21.15.16:0/1141996248 >> v2:172.21.15.156:3300/0 conn(0x7fcfa40d1fa0 msgr2=0x7fcfa40d23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.130+0000 7fcfa2ffd640 1 --2- 172.21.15.16:0/1141996248 >> v2:172.21.15.156:3300/0 conn(0x7fcfa40d1fa0 0x7fcfa40d23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.130+0000 7fcfa2ffd640 1 -- 172.21.15.16:0/1141996248 >> v1:172.21.15.16:6789/0 conn(0x7fcfa40b2bb0 legacy=0x7fcfa40cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.130 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.130+0000 7fcfa2ffd640 1 -- 172.21.15.16:0/1141996248 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa406c580 con 0x7fcfa415e020 2024-04-24T23:50:07.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.130+0000 7fcfa2ffd640 1 --2- 172.21.15.16:0/1141996248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa415e020 0x7fcfa40cf270 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcfa40c78f0 tx=0x7fcf9c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ad7ac7fdfdf82 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:07.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.131+0000 7fcfa1ffb640 1 -- 172.21.15.16:0/1141996248 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c037070 con 0x7fcfa415e020 2024-04-24T23:50:07.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.131+0000 7fcfa1ffb640 1 -- 172.21.15.16:0/1141996248 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf9c032440 con 0x7fcfa415e020 2024-04-24T23:50:07.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.131+0000 7fcfa1ffb640 1 -- 172.21.15.16:0/1141996248 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf9c02ae60 con 0x7fcfa415e020 2024-04-24T23:50:07.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.131+0000 7fcfa9431640 1 -- 172.21.15.16:0/1141996248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa415e020 msgr2=0x7fcfa40cf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.131+0000 7fcfa9431640 1 --2- 172.21.15.16:0/1141996248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa415e020 0x7fcfa40cf270 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcfa40c78f0 tx=0x7fcf9c02a110 comp rx=0 tx=0).stop 2024-04-24T23:50:07.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.131+0000 7fcfa9431640 1 -- 172.21.15.16:0/1141996248 shutdown_connections 2024-04-24T23:50:07.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.131+0000 7fcfa9431640 1 --2- 172.21.15.16:0/1141996248 >> v2:172.21.15.156:3300/0 conn(0x7fcfa40d1fa0 0x7fcfa40d23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.138 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.131+0000 7fcfa9431640 1 --2- 172.21.15.16:0/1141996248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa415e020 0x7fcfa40cf270 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.131+0000 7fcfa9431640 1 -- 172.21.15.16:0/1141996248 >> 172.21.15.16:0/1141996248 conn(0x7fcfa40d3e10 msgr2=0x7fcfa40d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.131+0000 7fcfa9431640 1 -- 172.21.15.16:0/1141996248 shutdown_connections 2024-04-24T23:50:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.131+0000 7fcfa9431640 1 -- 172.21.15.16:0/1141996248 wait complete. 2024-04-24T23:50:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.132+0000 7fcfa9431640 1 Processor -- start 2024-04-24T23:50:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.133+0000 7fcfa9431640 1 -- start start 2024-04-24T23:50:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.133+0000 7fcfa9431640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfa40b2bb0 0x7fcfa41f4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.133+0000 7fcfa9431640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa40d1fa0 0x7fcfa40cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.133+0000 7fcfa9431640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcfa415e020 0x7fcfa41f75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.133+0000 7fcfa9431640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa415e400 con 0x7fcfa40d1fa0 2024-04-24T23:50:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.133+0000 7fcfa9431640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa40d1a80 con 0x7fcfa40b2bb0 2024-04-24T23:50:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.133+0000 7fcfa9431640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fcfa40cfec0 con 0x7fcfa415e020 2024-04-24T23:50:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.133+0000 7fcfa27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa40d1fa0 0x7fcfa40cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.133+0000 7fcfa27fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa40d1fa0 0x7fcfa40cb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50510/0 (socket says 172.21.15.16:50510) 2024-04-24T23:50:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.133+0000 7fcfa27fc640 1 -- 172.21.15.16:0/2116171317 learned_addr learned my addr 172.21.15.16:0/2116171317 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcfa27fc640 1 -- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcfa415e020 msgr2=0x7fcfa41f75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcfa2ffd640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfa40b2bb0 0x7fcfa41f4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcfa37fe640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcfa415e020 0x7fcfa41f75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcfa27fc640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcfa415e020 0x7fcfa41f75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcfa27fc640 1 -- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfa40b2bb0 msgr2=0x7fcfa41f4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcfa27fc640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfa40b2bb0 0x7fcfa41f4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcfa27fc640 1 -- 172.21.15.16:0/2116171317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa41f7360 con 0x7fcfa40d1fa0 2024-04-24T23:50:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcfa2ffd640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfa40b2bb0 0x7fcfa41f4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcfa37fe640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcfa415e020 0x7fcfa41f75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcfa27fc640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa40d1fa0 0x7fcfa40cb8c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800c400 tx=0x7fcf9800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcf8b7fe640 1 -- 172.21.15.16:0/2116171317 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98017070 con 0x7fcfa40d1fa0 2024-04-24T23:50:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcf8b7fe640 1 -- 172.21.15.16:0/2116171317 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf98005ce0 con 0x7fcfa40d1fa0 2024-04-24T23:50:07.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcfa9431640 1 -- 172.21.15.16:0/2116171317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa40cc120 con 0x7fcfa40d1fa0 2024-04-24T23:50:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcf8b7fe640 1 -- 172.21.15.16:0/2116171317 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf980072f0 con 0x7fcfa40d1fa0 2024-04-24T23:50:07.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.134+0000 7fcfa9431640 1 -- 172.21.15.16:0/2116171317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa415e9e0 con 0x7fcfa40d1fa0 2024-04-24T23:50:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.135+0000 7fcfa9431640 1 -- 172.21.15.16:0/2116171317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfa40cfc60 con 0x7fcfa40d1fa0 2024-04-24T23:50:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.136+0000 7fcf8b7fe640 1 -- 172.21.15.16:0/2116171317 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcf9801a070 con 0x7fcfa40d1fa0 2024-04-24T23:50:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.136+0000 7fcf8b7fe640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcf9006be30 0x7fcf9006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.136+0000 7fcf8b7fe640 1 -- 172.21.15.16:0/2116171317 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fcf9808a690 con 0x7fcfa40d1fa0 2024-04-24T23:50:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.136+0000 7fcfa2ffd640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcf9006be30 0x7fcf9006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.137+0000 7fcfa2ffd640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcf9006be30 0x7fcf9006e2f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c034000 tx=0x7fcf9c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:07.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.140+0000 7fcf8b7fe640 1 -- 172.21.15.16:0/2116171317 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcfa40cfc60 con 0x7fcfa40d1fa0 2024-04-24T23:50:07.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.360+0000 7fcfa9431640 1 -- 172.21.15.16:0/2116171317 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfa406dfe0 con 0x7fcfa40d1fa0 2024-04-24T23:50:07.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.361+0000 7fcf8b7fe640 1 -- 172.21.15.16:0/2116171317 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+5180 (secure 0 0 0) 0x7fcfa41f7360 con 0x7fcfa40d1fa0 2024-04-24T23:50:07.360 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:07.361 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1623032341","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1623032341},{"type":"v1","addr":"172.21.15.156:6836","nonce":1623032341}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/3530738236","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":3530738236},{"type":"v1","addr":"172.21.15.16:6835","nonce":3530738236}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4260,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/2153063131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2153063131},{"type":"v1","addr":"172.21.15.16:6834","nonce":2153063131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:48:54.852787+0000","modified":"2024-04-24T23:49:17.914730+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4294},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.156:6837/289127714","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":289127714},{"type":"v1","addr":"172.21.15.156:6837","nonce":289127714}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4294,"qdb_cluster":[4294]},"id":1}]} 2024-04-24T23:50:07.361 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 8 2024-04-24T23:50:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.362+0000 7fcfa9431640 1 -- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcf9006be30 msgr2=0x7fcf9006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.362+0000 7fcfa9431640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcf9006be30 0x7fcf9006e2f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcf9c034000 tx=0x7fcf9c02e000 comp rx=0 tx=0).stop 2024-04-24T23:50:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.362+0000 7fcfa9431640 1 -- 172.21.15.16:0/2116171317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa40d1fa0 msgr2=0x7fcfa40cb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.362+0000 7fcfa9431640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa40d1fa0 0x7fcfa40cb8c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800c400 tx=0x7fcf9800c8c0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.363+0000 7fcfa9431640 1 -- 172.21.15.16:0/2116171317 shutdown_connections 2024-04-24T23:50:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.363+0000 7fcfa9431640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcf9006be30 0x7fcf9006e2f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.363+0000 7fcfa9431640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcfa415e020 0x7fcfa41f75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.363+0000 7fcfa9431640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfa40d1fa0 0x7fcfa40cb8c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.363+0000 7fcfa9431640 1 --2- 172.21.15.16:0/2116171317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcfa40b2bb0 0x7fcfa41f4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.363+0000 7fcfa9431640 1 -- 172.21.15.16:0/2116171317 >> 172.21.15.16:0/2116171317 conn(0x7fcfa40d3e10 msgr2=0x7fcfa40b40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.363+0000 7fcfa9431640 1 -- 172.21.15.16:0/2116171317 shutdown_connections 2024-04-24T23:50:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.363+0000 7fcfa9431640 1 -- 172.21.15.16:0/2116171317 wait complete. 2024-04-24T23:50:07.373 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:50:07.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.474+0000 7f338e0b3640 1 Processor -- start 2024-04-24T23:50:07.474 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.475+0000 7f338e0b3640 1 -- start start 2024-04-24T23:50:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.475+0000 7f338e0b3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f338815df20 0x7f3388071a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.475+0000 7f338e0b3640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f33880cfe50 0x7f3388071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.475+0000 7f338e0b3640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33880b6a70 con 0x7f33880d0530 2024-04-24T23:50:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.475+0000 7f338e0b3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f33880cf460 con 0x7f338815df20 2024-04-24T23:50:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.475+0000 7f338e0b3640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f33880b6820 con 0x7f33880cfe50 2024-04-24T23:50:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.475+0000 7f33877fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f338815df20 0x7f3388071a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.476+0000 7f33877fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f338815df20 0x7f3388071a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50526/0 (socket says 172.21.15.16:50526) 2024-04-24T23:50:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.476+0000 7f33877fe640 1 -- 172.21.15.16:0/3953959994 learned_addr learned my addr 172.21.15.16:0/3953959994 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.476+0000 7f3387fff640 1 --2- 172.21.15.16:0/3953959994 >> v2:172.21.15.156:3300/0 conn(0x7f33880cfe50 0x7f3388071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.476+0000 7f33877fe640 1 -- 172.21.15.16:0/3953959994 >> v2:172.21.15.156:3300/0 conn(0x7f33880cfe50 msgr2=0x7f3388071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.476+0000 7f33877fe640 1 --2- 172.21.15.16:0/3953959994 >> v2:172.21.15.156:3300/0 conn(0x7f33880cfe50 0x7f3388071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.476+0000 7f33877fe640 1 -- 172.21.15.16:0/3953959994 >> v1:172.21.15.156:6789/0 conn(0x7f33880d0530 legacy=0x7f33880ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:07.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.476+0000 7f33877fe640 1 -- 172.21.15.16:0/3953959994 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33880ce6a0 con 0x7f338815df20 2024-04-24T23:50:07.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.477+0000 7f33877fe640 1 --2- 172.21.15.16:0/3953959994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f338815df20 0x7f3388071a00 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f33880b65c0 tx=0x7f337802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48537d29212f79a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:07.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.477+0000 7f3386ffd640 1 -- 172.21.15.16:0/3953959994 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3378037070 con 0x7f338815df20 2024-04-24T23:50:07.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.477+0000 7f3386ffd640 1 -- 172.21.15.16:0/3953959994 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3378032440 con 0x7f338815df20 2024-04-24T23:50:07.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.477+0000 7f3386ffd640 1 -- 172.21.15.16:0/3953959994 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3378032810 con 0x7f338815df20 2024-04-24T23:50:07.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.477+0000 7f338e0b3640 1 -- 172.21.15.16:0/3953959994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f338815df20 msgr2=0x7f3388071a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.477+0000 7f338e0b3640 1 --2- 172.21.15.16:0/3953959994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f338815df20 0x7f3388071a00 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f33880b65c0 tx=0x7f337802a230 comp rx=0 tx=0).stop 2024-04-24T23:50:07.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.477+0000 7f338e0b3640 1 -- 172.21.15.16:0/3953959994 shutdown_connections 2024-04-24T23:50:07.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.477+0000 7f338e0b3640 1 --2- 172.21.15.16:0/3953959994 >> v2:172.21.15.156:3300/0 conn(0x7f33880cfe50 0x7f3388071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.477+0000 7f338e0b3640 1 --2- 172.21.15.16:0/3953959994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f338815df20 0x7f3388071a00 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.477+0000 7f338e0b3640 1 -- 172.21.15.16:0/3953959994 >> 172.21.15.16:0/3953959994 conn(0x7f33880d0ac0 msgr2=0x7f33880d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:07.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.477+0000 7f338e0b3640 1 -- 172.21.15.16:0/3953959994 shutdown_connections 2024-04-24T23:50:07.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.477+0000 7f338e0b3640 1 -- 172.21.15.16:0/3953959994 wait complete. 2024-04-24T23:50:07.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.478+0000 7f338e0b3640 1 Processor -- start 2024-04-24T23:50:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.479+0000 7f338e0b3640 1 -- start start 2024-04-24T23:50:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.479+0000 7f338e0b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33880cfe50 0x7f338806e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.479+0000 7f338e0b3640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33880d0530 0x7f3388150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.479+0000 7f338e0b3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f338815df20 0x7f33881568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.479+0000 7f338e0b3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f33880cef40 con 0x7f33880cfe50 2024-04-24T23:50:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.479+0000 7f338e0b3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f33880cec60 con 0x7f338815df20 2024-04-24T23:50:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.479+0000 7f338e0b3640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f33880b4030 con 0x7f33880d0530 2024-04-24T23:50:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f33877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33880cfe50 0x7f338806e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f33877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33880cfe50 0x7f338806e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50540/0 (socket says 172.21.15.16:50540) 2024-04-24T23:50:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f33877fe640 1 -- 172.21.15.16:0/4130479490 learned_addr learned my addr 172.21.15.16:0/4130479490 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f3387fff640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f338815df20 0x7f33881568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f337effd640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33880d0530 0x7f3388150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f33877fe640 1 -- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33880d0530 msgr2=0x7f3388150670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f33877fe640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33880d0530 0x7f3388150670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f33877fe640 1 -- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f338815df20 msgr2=0x7f33881568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f33877fe640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f338815df20 0x7f33881568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f33877fe640 1 -- 172.21.15.16:0/4130479490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f338806c420 con 0x7f33880cfe50 2024-04-24T23:50:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f3387fff640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f338815df20 0x7f33881568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f33877fe640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33880cfe50 0x7f338806e300 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3378000c00 tx=0x7f33780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f3384ff9640 1 -- 172.21.15.16:0/4130479490 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3378041020 con 0x7f33880cfe50 2024-04-24T23:50:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f3384ff9640 1 -- 172.21.15.16:0/4130479490 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33780306c0 con 0x7f33880cfe50 2024-04-24T23:50:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f3384ff9640 1 -- 172.21.15.16:0/4130479490 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3378037040 con 0x7f33880cfe50 2024-04-24T23:50:07.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f338e0b3640 1 -- 172.21.15.16:0/4130479490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f338806c700 con 0x7f33880cfe50 2024-04-24T23:50:07.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.480+0000 7f338e0b3640 1 -- 172.21.15.16:0/4130479490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f338806ccc0 con 0x7f33880cfe50 2024-04-24T23:50:07.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.481+0000 7f338e0b3640 1 -- 172.21.15.16:0/4130479490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3388068350 con 0x7f33880cfe50 2024-04-24T23:50:07.483 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.482+0000 7f3384ff9640 1 -- 172.21.15.16:0/4130479490 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3378044020 con 0x7f33880cfe50 2024-04-24T23:50:07.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.482+0000 7f3384ff9640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f336806be30 0x7f336806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.482+0000 7f3384ff9640 1 -- 172.21.15.16:0/4130479490 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7f33780aba10 con 0x7f33880cfe50 2024-04-24T23:50:07.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.482+0000 7f337effd640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f336806be30 0x7f336806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.483+0000 7f337effd640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f336806be30 0x7f336806e2f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3388156890 tx=0x7f3370009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:07.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.486+0000 7f3384ff9640 1 -- 172.21.15.16:0/4130479490 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3388068350 con 0x7f33880cfe50 2024-04-24T23:50:07.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.673+0000 7f338e0b3640 1 -- 172.21.15.16:0/4130479490 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f338806f8f0 con 0x7f33880cfe50 2024-04-24T23:50:07.674 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.675+0000 7f3384ff9640 1 -- 172.21.15.16:0/4130479490 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+18890 (secure 0 0 0) 0x7f337804b030 con 0x7f33880cfe50 2024-04-24T23:50:07.675 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:07.675 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:49:36.610962+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:48:53.157933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T23:48:53.823351+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-24T23:48:57.531302+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":7,"to":2}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]},{"pgid":"3.36","mappings":[{"from":5,"to":2}]},{"pgid":"3.38","mappings":[{"from":3,"to":1}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:50:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.677+0000 7f338e0b3640 1 -- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f336806be30 msgr2=0x7f336806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.677+0000 7f338e0b3640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f336806be30 0x7f336806e2f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3388156890 tx=0x7f3370009040 comp rx=0 tx=0).stop 2024-04-24T23:50:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.677+0000 7f338e0b3640 1 -- 172.21.15.16:0/4130479490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33880cfe50 msgr2=0x7f338806e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.677+0000 7f338e0b3640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33880cfe50 0x7f338806e300 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3378000c00 tx=0x7f33780046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.677+0000 7f338e0b3640 1 -- 172.21.15.16:0/4130479490 shutdown_connections 2024-04-24T23:50:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.677+0000 7f338e0b3640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f336806be30 0x7f336806e2f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.677+0000 7f338e0b3640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f338815df20 0x7f33881568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.677+0000 7f338e0b3640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33880d0530 0x7f3388150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.677+0000 7f338e0b3640 1 --2- 172.21.15.16:0/4130479490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33880cfe50 0x7f338806e300 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.677+0000 7f338e0b3640 1 -- 172.21.15.16:0/4130479490 >> 172.21.15.16:0/4130479490 conn(0x7f33880d0ac0 msgr2=0x7f3388160810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.677+0000 7f338e0b3640 1 -- 172.21.15.16:0/4130479490 shutdown_connections 2024-04-24T23:50:07.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.677+0000 7f338e0b3640 1 -- 172.21.15.16:0/4130479490 wait complete. 2024-04-24T23:50:07.688 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T23:50:07.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.791+0000 7fdd48e15640 1 Processor -- start 2024-04-24T23:50:07.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.791+0000 7fdd48e15640 1 -- start start 2024-04-24T23:50:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd48e15640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fdd44144740 0x7fdd4415c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd48e15640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd440cf3a0 con 0x7fdd4415c8e0 2024-04-24T23:50:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd48e15640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd440cc8d0 con 0x7fdd44071a00 2024-04-24T23:50:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd48e15640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fdd4415c1f0 con 0x7fdd44144740 2024-04-24T23:50:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd42d9d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdd4415c8e0 0x7fdd440ce7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58998/0 (socket says 172.21.15.16:58998) 2024-04-24T23:50:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd42d9d640 1 -- 172.21.15.16:0/452493948 learned_addr learned my addr 172.21.15.16:0/452493948 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd39d9b640 1 --2- 172.21.15.16:0/452493948 >> v2:172.21.15.156:3300/0 conn(0x7fdd44144740 0x7fdd4415c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 -- 172.21.15.16:0/452493948 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1474216101 0 0) 0x7fdd440cf3a0 con 0x7fdd4415c8e0 2024-04-24T23:50:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 -- 172.21.15.16:0/452493948 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd30003520 con 0x7fdd4415c8e0 2024-04-24T23:50:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 -- 172.21.15.16:0/452493948 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3708823274 0 0) 0x7fdd30003520 con 0x7fdd4415c8e0 2024-04-24T23:50:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 -- 172.21.15.16:0/452493948 >> v2:172.21.15.156:3300/0 conn(0x7fdd44144740 msgr2=0x7fdd4415c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 --2- 172.21.15.16:0/452493948 >> v2:172.21.15.156:3300/0 conn(0x7fdd44144740 0x7fdd4415c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 -- 172.21.15.16:0/452493948 >> v1:172.21.15.156:6789/0 conn(0x7fdd44071a00 legacy=0x7fdd44071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 -- 172.21.15.16:0/452493948 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd440ccbb0 con 0x7fdd4415c8e0 2024-04-24T23:50:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 -- 172.21.15.16:0/452493948 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7fdd3c0019f0 con 0x7fdd4415c8e0 2024-04-24T23:50:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 -- 172.21.15.16:0/452493948 >> v1:172.21.15.16:6789/0 conn(0x7fdd4415c8e0 legacy=0x7fdd440ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 --2- 172.21.15.16:0/452493948 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd300039b0 0x7fdd30003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 --2- 172.21.15.16:0/452493948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd30004800 0x7fdd30004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 --2- 172.21.15.16:0/452493948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd300051c0 0x7fdd3000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 -- 172.21.15.16:0/452493948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3000c030 con 0x7fdd300051c0 2024-04-24T23:50:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 -- 172.21.15.16:0/452493948 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdd3000c1e0 con 0x7fdd30004800 2024-04-24T23:50:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd41d9b640 1 -- 172.21.15.16:0/452493948 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fdd3000c3e0 con 0x7fdd300039b0 2024-04-24T23:50:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.792+0000 7fdd42d9d640 1 --2- 172.21.15.16:0/452493948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd300051c0 0x7fdd3000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.793+0000 7fdd4259c640 1 --2- 172.21.15.16:0/452493948 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd300039b0 0x7fdd30003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.793+0000 7fdd39d9b640 1 --2- 172.21.15.16:0/452493948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd30004800 0x7fdd30004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.793+0000 7fdd42d9d640 1 -- 172.21.15.16:0/452493948 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd300039b0 msgr2=0x7fdd30003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.793+0000 7fdd42d9d640 1 --2- 172.21.15.16:0/452493948 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd300039b0 0x7fdd30003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.793+0000 7fdd42d9d640 1 -- 172.21.15.16:0/452493948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd30004800 msgr2=0x7fdd30004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.793+0000 7fdd42d9d640 1 --2- 172.21.15.16:0/452493948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd30004800 0x7fdd30004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.793+0000 7fdd42d9d640 1 -- 172.21.15.16:0/452493948 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd30003520 con 0x7fdd300051c0 2024-04-24T23:50:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.793+0000 7fdd39d9b640 1 --2- 172.21.15.16:0/452493948 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd30004800 0x7fdd30004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.793+0000 7fdd42d9d640 1 -- 172.21.15.16:0/452493948 reap_dead start 2024-04-24T23:50:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.793+0000 7fdd42d9d640 1 --2- 172.21.15.16:0/452493948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd300051c0 0x7fdd3000b9c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00a420 tx=0x7fdd30005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adad4bece1f4f8d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.794+0000 7fdd41d9b640 1 -- 172.21.15.16:0/452493948 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c00a460 con 0x7fdd300051c0 2024-04-24T23:50:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.794+0000 7fdd41d9b640 1 -- 172.21.15.16:0/452493948 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd3c031070 con 0x7fdd300051c0 2024-04-24T23:50:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.794+0000 7fdd41d9b640 1 -- 172.21.15.16:0/452493948 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3c036470 con 0x7fdd300051c0 2024-04-24T23:50:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.794+0000 7fdd48e15640 1 -- 172.21.15.16:0/452493948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd300051c0 msgr2=0x7fdd3000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.794+0000 7fdd48e15640 1 --2- 172.21.15.16:0/452493948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd300051c0 0x7fdd3000b9c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fdd3c00a420 tx=0x7fdd30005830 comp rx=0 tx=0).stop 2024-04-24T23:50:07.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.794+0000 7fdd48e15640 1 -- 172.21.15.16:0/452493948 shutdown_connections 2024-04-24T23:50:07.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.794+0000 7fdd48e15640 1 --2- 172.21.15.16:0/452493948 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd300051c0 0x7fdd3000b9c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.794+0000 7fdd48e15640 1 -- 172.21.15.16:0/452493948 >> 172.21.15.16:0/452493948 conn(0x7fdd440b2000 msgr2=0x7fdd440cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:07.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.794+0000 7fdd48e15640 1 -- 172.21.15.16:0/452493948 shutdown_connections 2024-04-24T23:50:07.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.794+0000 7fdd48e15640 1 -- 172.21.15.16:0/452493948 wait complete. 2024-04-24T23:50:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.796+0000 7fdd48e15640 1 Processor -- start 2024-04-24T23:50:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.796+0000 7fdd48e15640 1 -- start start 2024-04-24T23:50:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.796+0000 7fdd48e15640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd300051c0 0x7fdd440ad6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.796+0000 7fdd48e15640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd440ae210 0x7fdd440ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.796+0000 7fdd48e15640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd440a95d0 0x7fdd440a9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.796+0000 7fdd48e15640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4415c3a0 con 0x7fdd300051c0 2024-04-24T23:50:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.796+0000 7fdd48e15640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdd440cd170 con 0x7fdd440ae210 2024-04-24T23:50:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.796+0000 7fdd48e15640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fdd4415c520 con 0x7fdd440a95d0 2024-04-24T23:50:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.796+0000 7fdd4259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd300051c0 0x7fdd440ad6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd4259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd300051c0 0x7fdd440ad6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50552/0 (socket says 172.21.15.16:50552) 2024-04-24T23:50:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd4259c640 1 -- 172.21.15.16:0/3846933711 learned_addr learned my addr 172.21.15.16:0/3846933711 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd42d9d640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd440a95d0 0x7fdd440a9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd39d9b640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd440ae210 0x7fdd440ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd4259c640 1 -- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd440a95d0 msgr2=0x7fdd440a9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd4259c640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd440a95d0 0x7fdd440a9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd4259c640 1 -- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd440ae210 msgr2=0x7fdd440ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd4259c640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd440ae210 0x7fdd440ae670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd4259c640 1 -- 172.21.15.16:0/3846933711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd440aa370 con 0x7fdd300051c0 2024-04-24T23:50:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd39d9b640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd440ae210 0x7fdd440ae670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd4259c640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd300051c0 0x7fdd440ad6b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400f5f0 tx=0x7fdd3400fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd3b7fe640 1 -- 172.21.15.16:0/3846933711 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd340052e0 con 0x7fdd300051c0 2024-04-24T23:50:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd3b7fe640 1 -- 172.21.15.16:0/3846933711 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd34018070 con 0x7fdd300051c0 2024-04-24T23:50:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd3b7fe640 1 -- 172.21.15.16:0/3846933711 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd34003810 con 0x7fdd300051c0 2024-04-24T23:50:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd48e15640 1 -- 172.21.15.16:0/3846933711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd440aa650 con 0x7fdd300051c0 2024-04-24T23:50:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.797+0000 7fdd48e15640 1 -- 172.21.15.16:0/3846933711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd440aac10 con 0x7fdd300051c0 2024-04-24T23:50:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.798+0000 7fdd48e15640 1 -- 172.21.15.16:0/3846933711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd440aaff0 con 0x7fdd300051c0 2024-04-24T23:50:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.799+0000 7fdd3b7fe640 1 -- 172.21.15.16:0/3846933711 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdd340039b0 con 0x7fdd300051c0 2024-04-24T23:50:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.799+0000 7fdd3b7fe640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdd2406be30 0x7fdd2406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.799+0000 7fdd3b7fe640 1 -- 172.21.15.16:0/3846933711 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 6735+0+0 (secure 0 0 0) 0x7fdd3408afa0 con 0x7fdd300051c0 2024-04-24T23:50:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.799+0000 7fdd39d9b640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdd2406be30 0x7fdd2406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.800+0000 7fdd39d9b640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdd2406be30 0x7fdd2406e2f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdd440ae0a0 tx=0x7fdd2c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:07.803 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:07.803+0000 7fdd3b7fe640 1 -- 172.21.15.16:0/3846933711 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fdd440aaff0 con 0x7fdd300051c0 2024-04-24T23:50:08.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.030+0000 7fdd48e15640 1 -- 172.21.15.16:0/3846933711 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fdd440731f0 con 0x7fdd300051c0 2024-04-24T23:50:08.030 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-04-24T23:50:08.030+0000 7fa77c6c1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T23:50:08.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.032+0000 7fdd3b7fe640 1 -- 172.21.15.16:0/3846933711 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) ==== 161+0+0 (secure 0 0 0) 0x7fdd34059550 con 0x7fdd300051c0 2024-04-24T23:50:08.031 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T23:50:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.035+0000 7fdd48e15640 1 -- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdd2406be30 msgr2=0x7fdd2406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.035+0000 7fdd48e15640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdd2406be30 0x7fdd2406e2f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fdd440ae0a0 tx=0x7fdd2c008040 comp rx=0 tx=0).stop 2024-04-24T23:50:08.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.035+0000 7fdd48e15640 1 -- 172.21.15.16:0/3846933711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd300051c0 msgr2=0x7fdd440ad6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:08.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.035+0000 7fdd48e15640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd300051c0 0x7fdd440ad6b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fdd3400f5f0 tx=0x7fdd3400fab0 comp rx=0 tx=0).stop 2024-04-24T23:50:08.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.035+0000 7fdd48e15640 1 -- 172.21.15.16:0/3846933711 shutdown_connections 2024-04-24T23:50:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.035+0000 7fdd48e15640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fdd2406be30 0x7fdd2406e2f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.035+0000 7fdd48e15640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fdd440a95d0 0x7fdd440a9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.035+0000 7fdd48e15640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdd440ae210 0x7fdd440ae670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.035+0000 7fdd48e15640 1 --2- 172.21.15.16:0/3846933711 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdd300051c0 0x7fdd440ad6b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:08.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.035+0000 7fdd48e15640 1 -- 172.21.15.16:0/3846933711 >> 172.21.15.16:0/3846933711 conn(0x7fdd440b2000 msgr2=0x7fdd440709f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:08.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.036+0000 7fdd48e15640 1 -- 172.21.15.16:0/3846933711 shutdown_connections 2024-04-24T23:50:08.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.036+0000 7fdd48e15640 1 -- 172.21.15.16:0/3846933711 wait complete. 2024-04-24T23:50:08.039 INFO:tasks.ceph.mds.b.smithi156.stderr: -4691> 2024-04-24T23:49:00.633+0000 7f4c04643640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T23:50:08.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T23:50:08.070 INFO:tasks.ceph.mds.b.smithi156.stdout:starting mds.b at 2024-04-24T23:50:08.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.149+0000 7f20d9aa5640 1 Processor -- start 2024-04-24T23:50:08.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.149+0000 7f20d9aa5640 1 -- start start 2024-04-24T23:50:08.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.150+0000 7f20d9aa5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f20d406bc40 0x7f20d406c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:08.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.150+0000 7f20d9aa5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20d40b7050 0x7f20d40b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:08.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.150+0000 7f20d9aa5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20d40d24d0 con 0x7f20d415dfd0 2024-04-24T23:50:08.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.150+0000 7f20d9aa5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f20d406ba00 con 0x7f20d406bc40 2024-04-24T23:50:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.150+0000 7f20d9aa5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f20d40d27b0 con 0x7f20d40b7050 2024-04-24T23:50:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.150+0000 7f20d37fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f20d415dfd0 0x7f20d406b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59000/0 (socket says 172.21.15.16:59000) 2024-04-24T23:50:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.150+0000 7f20d37fe640 1 -- 172.21.15.16:0/1918486854 learned_addr learned my addr 172.21.15.16:0/1918486854 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d3fff640 1 --2- 172.21.15.16:0/1918486854 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20d40b7050 0x7f20d40b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d2ffd640 1 --2- 172.21.15.16:0/1918486854 >> v2:172.21.15.156:3300/0 conn(0x7f20d406bc40 0x7f20d406c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 -- 172.21.15.16:0/1918486854 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161590427 0 0) 0x7f20d40d24d0 con 0x7f20d415dfd0 2024-04-24T23:50:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 -- 172.21.15.16:0/1918486854 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20bc003520 con 0x7f20d415dfd0 2024-04-24T23:50:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 -- 172.21.15.16:0/1918486854 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1611593012 0 0) 0x7f20bc003520 con 0x7f20d415dfd0 2024-04-24T23:50:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 -- 172.21.15.16:0/1918486854 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20d40b7050 msgr2=0x7f20d40b74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 --2- 172.21.15.16:0/1918486854 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20d40b7050 0x7f20d40b74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 -- 172.21.15.16:0/1918486854 >> v2:172.21.15.156:3300/0 conn(0x7f20d406bc40 msgr2=0x7f20d406c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 --2- 172.21.15.16:0/1918486854 >> v2:172.21.15.156:3300/0 conn(0x7f20d406bc40 0x7f20d406c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 -- 172.21.15.16:0/1918486854 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d40d2a90 con 0x7f20d415dfd0 2024-04-24T23:50:08.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 -- 172.21.15.16:0/1918486854 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f20c4001670 con 0x7f20d415dfd0 2024-04-24T23:50:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 -- 172.21.15.16:0/1918486854 >> v1:172.21.15.16:6789/0 conn(0x7f20d415dfd0 legacy=0x7f20d406b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 --2- 172.21.15.16:0/1918486854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20bc0039b0 0x7f20bc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 --2- 172.21.15.16:0/1918486854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20bc004800 0x7f20bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 --2- 172.21.15.16:0/1918486854 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20d40b7050 0x7f20bc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 -- 172.21.15.16:0/1918486854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f20d406ba00 con 0x7f20bc0039b0 2024-04-24T23:50:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 -- 172.21.15.16:0/1918486854 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f20d40d27b0 con 0x7f20bc004800 2024-04-24T23:50:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d27fc640 1 -- 172.21.15.16:0/1918486854 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f20bc00b9c0 con 0x7f20d40b7050 2024-04-24T23:50:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d37fe640 1 --2- 172.21.15.16:0/1918486854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20bc0039b0 0x7f20bc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d2ffd640 1 --2- 172.21.15.16:0/1918486854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20bc004800 0x7f20bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d3fff640 1 --2- 172.21.15.16:0/1918486854 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20d40b7050 0x7f20bc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d37fe640 1 -- 172.21.15.16:0/1918486854 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20d40b7050 msgr2=0x7f20bc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d37fe640 1 --2- 172.21.15.16:0/1918486854 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20d40b7050 0x7f20bc0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:08.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d37fe640 1 -- 172.21.15.16:0/1918486854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20bc004800 msgr2=0x7f20bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:08.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d37fe640 1 --2- 172.21.15.16:0/1918486854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20bc004800 0x7f20bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:08.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d37fe640 1 -- 172.21.15.16:0/1918486854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20bc003520 con 0x7f20bc0039b0 2024-04-24T23:50:08.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.151+0000 7f20d2ffd640 1 --2- 172.21.15.16:0/1918486854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20bc004800 0x7f20bc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:08.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.152+0000 7f20d37fe640 1 --2- 172.21.15.16:0/1918486854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20bc0039b0 0x7f20bc003dd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f20c400c270 tx=0x7f20bc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97f10e27c5eef6f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:08.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.152+0000 7f20d27fc640 1 -- 172.21.15.16:0/1918486854 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c4007d30 con 0x7f20bc0039b0 2024-04-24T23:50:08.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.152+0000 7f20d27fc640 1 -- 172.21.15.16:0/1918486854 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c4005ad0 con 0x7f20bc0039b0 2024-04-24T23:50:08.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.152+0000 7f20d27fc640 1 -- 172.21.15.16:0/1918486854 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c4007070 con 0x7f20bc0039b0 2024-04-24T23:50:08.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.153+0000 7f20d9aa5640 1 -- 172.21.15.16:0/1918486854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20bc0039b0 msgr2=0x7f20bc003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:08.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.153+0000 7f20d9aa5640 1 --2- 172.21.15.16:0/1918486854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20bc0039b0 0x7f20bc003dd0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f20c400c270 tx=0x7f20bc005830 comp rx=0 tx=0).stop 2024-04-24T23:50:08.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.153+0000 7f20d3fff640 1 -- 172.21.15.16:0/1918486854 reap_dead start 2024-04-24T23:50:08.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.153+0000 7f20d9aa5640 1 -- 172.21.15.16:0/1918486854 shutdown_connections 2024-04-24T23:50:08.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.153+0000 7f20d9aa5640 1 -- 172.21.15.16:0/1918486854 >> 172.21.15.16:0/1918486854 conn(0x7f20d40d3dc0 msgr2=0x7f20d40d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:08.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.153+0000 7f20d9aa5640 1 -- 172.21.15.16:0/1918486854 shutdown_connections 2024-04-24T23:50:08.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.153+0000 7f20d9aa5640 1 -- 172.21.15.16:0/1918486854 wait complete. 2024-04-24T23:50:08.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.154+0000 7f20d9aa5640 1 Processor -- start 2024-04-24T23:50:08.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.154+0000 7f20d9aa5640 1 -- start start 2024-04-24T23:50:08.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d9aa5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20bc0039b0 0x7f20d41e8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:08.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d9aa5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20d40cdc80 0x7f20d414ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:08.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d9aa5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20d41e76b0 0x7f20d406c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:08.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d9aa5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f20d40cead0 con 0x7f20d40cdc80 2024-04-24T23:50:08.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d9aa5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f20d40d3050 con 0x7f20bc0039b0 2024-04-24T23:50:08.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d9aa5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f20d40b74a0 con 0x7f20d41e76b0 2024-04-24T23:50:08.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20d40cdc80 0x7f20d414ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:08.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d2ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20d40cdc80 0x7f20d414ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50572/0 (socket says 172.21.15.16:50572) 2024-04-24T23:50:08.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d2ffd640 1 -- 172.21.15.16:0/3476436547 learned_addr learned my addr 172.21.15.16:0/3476436547 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:08.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d3fff640 1 --2- 172.21.15.16:0/3476436547 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20d41e76b0 0x7f20d406c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:08.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d37fe640 1 --2- 172.21.15.16:0/3476436547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20bc0039b0 0x7f20d41e8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:08.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d2ffd640 1 -- 172.21.15.16:0/3476436547 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20d41e76b0 msgr2=0x7f20d406c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:08.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d2ffd640 1 --2- 172.21.15.16:0/3476436547 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20d41e76b0 0x7f20d406c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:08.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d2ffd640 1 -- 172.21.15.16:0/3476436547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20bc0039b0 msgr2=0x7f20d41e8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:08.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d2ffd640 1 --2- 172.21.15.16:0/3476436547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20bc0039b0 0x7f20d41e8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:08.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.155+0000 7f20d2ffd640 1 -- 172.21.15.16:0/3476436547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20d40ce0e0 con 0x7f20d40cdc80 2024-04-24T23:50:08.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.156+0000 7f20d2ffd640 1 --2- 172.21.15.16:0/3476436547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20d40cdc80 0x7f20d414ef50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f20c8011570 tx=0x7f20c8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:08.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.156+0000 7f20bbfff640 1 -- 172.21.15.16:0/3476436547 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c8015870 con 0x7f20d40cdc80 2024-04-24T23:50:08.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.156+0000 7f20bbfff640 1 -- 172.21.15.16:0/3476436547 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20c8015a10 con 0x7f20d40cdc80 2024-04-24T23:50:08.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.156+0000 7f20d9aa5640 1 -- 172.21.15.16:0/3476436547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20d41e6610 con 0x7f20d40cdc80 2024-04-24T23:50:08.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.156+0000 7f20bbfff640 1 -- 172.21.15.16:0/3476436547 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20c8015dc0 con 0x7f20d40cdc80 2024-04-24T23:50:08.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.156+0000 7f20d9aa5640 1 -- 172.21.15.16:0/3476436547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20d41e6bd0 con 0x7f20d40cdc80 2024-04-24T23:50:08.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.157+0000 7f20d9aa5640 1 -- 172.21.15.16:0/3476436547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20d4068f90 con 0x7f20d40cdc80 2024-04-24T23:50:08.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.157+0000 7f20bbfff640 1 -- 172.21.15.16:0/3476436547 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f20c800f460 con 0x7f20d40cdc80 2024-04-24T23:50:08.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.158+0000 7f20bbfff640 1 --2- 172.21.15.16:0/3476436547 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f20c006be60 0x7f20c006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:08.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.158+0000 7f20bbfff640 1 -- 172.21.15.16:0/3476436547 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6821+0+0 (secure 0 0 0) 0x7f20c808bac0 con 0x7f20d40cdc80 2024-04-24T23:50:08.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.158+0000 7f20d37fe640 1 --2- 172.21.15.16:0/3476436547 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f20c006be60 0x7f20c006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:08.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.158+0000 7f20d37fe640 1 --2- 172.21.15.16:0/3476436547 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f20c006be60 0x7f20c006e320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f20c4033000 tx=0x7f20c4009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:08.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.162+0000 7f20bbfff640 1 -- 172.21.15.16:0/3476436547 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f20c8019030 con 0x7f20d40cdc80 2024-04-24T23:50:08.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:08.385+0000 7f20d9aa5640 1 -- 172.21.15.16:0/3476436547 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f20d40d1ed0 con 0x7f20d40cdc80 2024-04-24T23:50:09.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.033+0000 7f20bbfff640 1 -- 172.21.15.16:0/3476436547 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) ==== 104+0+0 (secure 0 0 0) 0x7f20c805a000 con 0x7f20d40cdc80 2024-04-24T23:50:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.036+0000 7f20d9aa5640 1 -- 172.21.15.16:0/3476436547 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f20c006be60 msgr2=0x7f20c006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.036+0000 7f20d9aa5640 1 --2- 172.21.15.16:0/3476436547 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f20c006be60 0x7f20c006e320 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f20c4033000 tx=0x7f20c4009030 comp rx=0 tx=0).stop 2024-04-24T23:50:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.036+0000 7f20d9aa5640 1 -- 172.21.15.16:0/3476436547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20d40cdc80 msgr2=0x7f20d414ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.036+0000 7f20d9aa5640 1 --2- 172.21.15.16:0/3476436547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20d40cdc80 0x7f20d414ef50 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f20c8011570 tx=0x7f20c8011a30 comp rx=0 tx=0).stop 2024-04-24T23:50:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.036+0000 7f20d9aa5640 1 -- 172.21.15.16:0/3476436547 shutdown_connections 2024-04-24T23:50:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.036+0000 7f20d9aa5640 1 --2- 172.21.15.16:0/3476436547 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f20c006be60 0x7f20c006e320 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.036+0000 7f20d9aa5640 1 --2- 172.21.15.16:0/3476436547 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f20d41e76b0 0x7f20d406c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.036+0000 7f20d9aa5640 1 --2- 172.21.15.16:0/3476436547 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20d40cdc80 0x7f20d414ef50 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:09.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.036+0000 7f20d9aa5640 1 --2- 172.21.15.16:0/3476436547 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f20bc0039b0 0x7f20d41e8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.036+0000 7f20d9aa5640 1 -- 172.21.15.16:0/3476436547 >> 172.21.15.16:0/3476436547 conn(0x7f20d40d3dc0 msgr2=0x7f20d41603c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.036+0000 7f20d9aa5640 1 -- 172.21.15.16:0/3476436547 shutdown_connections 2024-04-24T23:50:09.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.036+0000 7f20d9aa5640 1 -- 172.21.15.16:0/3476436547 wait complete. 2024-04-24T23:50:09.050 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-24T23:50:09.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.162+0000 7f6ee3509640 1 Processor -- start 2024-04-24T23:50:09.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.163+0000 7f6ee3509640 1 -- start start 2024-04-24T23:50:09.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.163+0000 7f6ee3509640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6edc15dfb0 0x7f6edc0cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:09.164 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.163+0000 7f6ee3509640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6edc06c110 0x7f6edc162250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.163+0000 7f6ee3509640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6edc0d1aa0 con 0x7f6edc06bcb0 2024-04-24T23:50:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.164+0000 7f6ee3509640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6edc0d1dd0 con 0x7f6edc06c110 2024-04-24T23:50:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.164+0000 7f6ee3509640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f6edc06b470 con 0x7f6edc15dfb0 2024-04-24T23:50:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.164+0000 7f6ee1aa6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6edc06c110 0x7f6edc162250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.164+0000 7f6ee0aa4640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6edc06bcb0 0x7f6edc161b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59006/0 (socket says 172.21.15.16:59006) 2024-04-24T23:50:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.164+0000 7f6ee0aa4640 1 -- 172.21.15.16:0/1893711030 learned_addr learned my addr 172.21.15.16:0/1893711030 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.164+0000 7f6ee12a5640 1 --2- 172.21.15.16:0/1893711030 >> v2:172.21.15.156:3300/0 conn(0x7f6edc15dfb0 0x7f6edc0cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.164+0000 7f6ee1aa6640 1 -- 172.21.15.16:0/1893711030 >> v2:172.21.15.156:3300/0 conn(0x7f6edc15dfb0 msgr2=0x7f6edc0cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.164+0000 7f6ee1aa6640 1 --2- 172.21.15.16:0/1893711030 >> v2:172.21.15.156:3300/0 conn(0x7f6edc15dfb0 0x7f6edc0cfba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.164+0000 7f6ee1aa6640 1 -- 172.21.15.16:0/1893711030 >> v1:172.21.15.16:6789/0 conn(0x7f6edc06bcb0 legacy=0x7f6edc161b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.164+0000 7f6edbfff640 1 -- 172.21.15.16:0/1893711030 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1102904330 0 0) 0x7f6edc0d1aa0 con 0x7f6edc06bcb0 2024-04-24T23:50:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.164+0000 7f6ee1aa6640 1 -- 172.21.15.16:0/1893711030 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6edc06b730 con 0x7f6edc06c110 2024-04-24T23:50:09.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.165+0000 7f6ee1aa6640 1 --2- 172.21.15.16:0/1893711030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc06c110 0x7f6edc162250 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f6edc0b69d0 tx=0x7f6ecc02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16f4f035f9b96acf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:09.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.165+0000 7f6edbfff640 1 -- 172.21.15.16:0/1893711030 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc007740 con 0x7f6edc06c110 2024-04-24T23:50:09.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.165+0000 7f6edbfff640 1 -- 172.21.15.16:0/1893711030 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ecc0078e0 con 0x7f6edc06c110 2024-04-24T23:50:09.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.165+0000 7f6edbfff640 1 -- 172.21.15.16:0/1893711030 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc007cb0 con 0x7f6edc06c110 2024-04-24T23:50:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.165+0000 7f6ee3509640 1 -- 172.21.15.16:0/1893711030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc06c110 msgr2=0x7f6edc162250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:09.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.165+0000 7f6ee3509640 1 --2- 172.21.15.16:0/1893711030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc06c110 0x7f6edc162250 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f6edc0b69d0 tx=0x7f6ecc02a680 comp rx=0 tx=0).stop 2024-04-24T23:50:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.165+0000 7f6ee3509640 1 -- 172.21.15.16:0/1893711030 shutdown_connections 2024-04-24T23:50:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.165+0000 7f6ee3509640 1 --2- 172.21.15.16:0/1893711030 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc06c110 0x7f6edc162250 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.165+0000 7f6ee3509640 1 --2- 172.21.15.16:0/1893711030 >> v2:172.21.15.156:3300/0 conn(0x7f6edc15dfb0 0x7f6edc0cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.165+0000 7f6ee3509640 1 -- 172.21.15.16:0/1893711030 >> 172.21.15.16:0/1893711030 conn(0x7f6edc0d3e30 msgr2=0x7f6edc0d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.166+0000 7f6ee3509640 1 -- 172.21.15.16:0/1893711030 shutdown_connections 2024-04-24T23:50:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.166+0000 7f6ee3509640 1 -- 172.21.15.16:0/1893711030 wait complete. 2024-04-24T23:50:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.167+0000 7f6ee3509640 1 Processor -- start 2024-04-24T23:50:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.167+0000 7f6ee3509640 1 -- start start 2024-04-24T23:50:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.167+0000 7f6ee3509640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6edc06bcb0 0x7f6edc1f81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.167+0000 7f6ee3509640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc06c110 0x7f6edc151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.167+0000 7f6ee3509640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6edc15dfb0 0x7f6edc15ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.167+0000 7f6ee3509640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc06c560 con 0x7f6edc06c110 2024-04-24T23:50:09.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.167+0000 7f6ee3509640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6edc06ba10 con 0x7f6edc15dfb0 2024-04-24T23:50:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.167+0000 7f6ee3509640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6edc0b6f60 con 0x7f6edc06bcb0 2024-04-24T23:50:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ee0aa4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc06c110 0x7f6edc151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ee0aa4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc06c110 0x7f6edc151c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50590/0 (socket says 172.21.15.16:50590) 2024-04-24T23:50:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ee0aa4640 1 -- 172.21.15.16:0/897498737 learned_addr learned my addr 172.21.15.16:0/897498737 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ee1aa6640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6edc15dfb0 0x7f6edc15ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ee12a5640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6edc06bcb0 0x7f6edc1f81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ee0aa4640 1 -- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6edc06bcb0 msgr2=0x7f6edc1f81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ee0aa4640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6edc06bcb0 0x7f6edc1f81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ee0aa4640 1 -- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6edc15dfb0 msgr2=0x7f6edc15ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ee0aa4640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6edc15dfb0 0x7f6edc15ac00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ee0aa4640 1 -- 172.21.15.16:0/897498737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6edc1f8700 con 0x7f6edc06c110 2024-04-24T23:50:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ee1aa6640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6edc15dfb0 0x7f6edc15ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:09.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ee0aa4640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc06c110 0x7f6edc151c30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000c430 tx=0x7f6ed000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:09.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ed9ffb640 1 -- 172.21.15.16:0/897498737 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed00074e0 con 0x7f6edc06c110 2024-04-24T23:50:09.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ed9ffb640 1 -- 172.21.15.16:0/897498737 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ed00047e0 con 0x7f6edc06c110 2024-04-24T23:50:09.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ee3509640 1 -- 172.21.15.16:0/897498737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6edc152bf0 con 0x7f6edc06c110 2024-04-24T23:50:09.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.168+0000 7f6ed9ffb640 1 -- 172.21.15.16:0/897498737 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ed0004bd0 con 0x7f6edc06c110 2024-04-24T23:50:09.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.169+0000 7f6ee3509640 1 -- 172.21.15.16:0/897498737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6edc152eb0 con 0x7f6edc06c110 2024-04-24T23:50:09.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.169+0000 7f6ee3509640 1 -- 172.21.15.16:0/897498737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6edc068f90 con 0x7f6edc06c110 2024-04-24T23:50:09.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.170+0000 7f6ed9ffb640 1 -- 172.21.15.16:0/897498737 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6ed0011070 con 0x7f6edc06c110 2024-04-24T23:50:09.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.170+0000 7f6ed9ffb640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6ec806be30 0x7f6ec806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:09.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.170+0000 7f6ed9ffb640 1 -- 172.21.15.16:0/897498737 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 6821+0+0 (secure 0 0 0) 0x7f6ed008ae50 con 0x7f6edc06c110 2024-04-24T23:50:09.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.171+0000 7f6ee12a5640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6ec806be30 0x7f6ec806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:09.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.171+0000 7f6ee12a5640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6ec806be30 0x7f6ec806e2f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6ed400aac0 tx=0x7f6ed4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:09.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.174+0000 7f6ed9ffb640 1 -- 172.21.15.16:0/897498737 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6edc1f8700 con 0x7f6edc06c110 2024-04-24T23:50:09.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:09.358+0000 7f6ee3509640 1 -- 172.21.15.16:0/897498737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f6edc0cecb0 con 0x7f6edc06c110 2024-04-24T23:50:10.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.037+0000 7f6ed9ffb640 1 -- 172.21.15.16:0/897498737 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) ==== 181+0+0 (secure 0 0 0) 0x7f6ed00593a0 con 0x7f6edc06c110 2024-04-24T23:50:10.037 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-04-24T23:50:10.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.044+0000 7f6ee3509640 1 -- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6ec806be30 msgr2=0x7f6ec806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:10.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.044+0000 7f6ee3509640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6ec806be30 0x7f6ec806e2f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f6ed400aac0 tx=0x7f6ed4008040 comp rx=0 tx=0).stop 2024-04-24T23:50:10.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.044+0000 7f6ee3509640 1 -- 172.21.15.16:0/897498737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc06c110 msgr2=0x7f6edc151c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:10.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.044+0000 7f6ee3509640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc06c110 0x7f6edc151c30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f6ed000c430 tx=0x7f6ed000c8f0 comp rx=0 tx=0).stop 2024-04-24T23:50:10.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.044+0000 7f6ee3509640 1 -- 172.21.15.16:0/897498737 shutdown_connections 2024-04-24T23:50:10.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.044+0000 7f6ee3509640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6ec806be30 0x7f6ec806e2f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:10.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.044+0000 7f6ee3509640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6edc15dfb0 0x7f6edc15ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:10.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.044+0000 7f6ee3509640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6edc06c110 0x7f6edc151c30 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:10.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.044+0000 7f6ee3509640 1 --2- 172.21.15.16:0/897498737 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6edc06bcb0 0x7f6edc1f81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:10.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.044+0000 7f6ee3509640 1 -- 172.21.15.16:0/897498737 >> 172.21.15.16:0/897498737 conn(0x7f6edc0d3e30 msgr2=0x7f6edc15bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:10.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.045+0000 7f6ee3509640 1 -- 172.21.15.16:0/897498737 shutdown_connections 2024-04-24T23:50:10.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.045+0000 7f6ee3509640 1 -- 172.21.15.16:0/897498737 wait complete. 2024-04-24T23:50:10.059 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-24T23:50:10.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.170+0000 7fd8a6ea5640 1 Processor -- start 2024-04-24T23:50:10.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.170+0000 7fd8a6ea5640 1 -- start start 2024-04-24T23:50:10.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.171+0000 7fd8a6ea5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd8a01439d0 0x7fd8a00cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:10.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.171+0000 7fd8a6ea5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd8a00ced10 0x7fd8a006c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:10.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.171+0000 7fd8a6ea5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8a00b3590 con 0x7fd8a00b3060 2024-04-24T23:50:10.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.171+0000 7fd8a6ea5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd8a0143db0 con 0x7fd8a01439d0 2024-04-24T23:50:10.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.171+0000 7fd8a6ea5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fd8a00b3870 con 0x7fd8a00ced10 2024-04-24T23:50:10.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.171+0000 7fd8a4c41640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd8a01439d0 0x7fd8a00cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:10.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.171+0000 7fd89ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd8a00b3060 0x7fd8a00ce600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59016/0 (socket says 172.21.15.16:59016) 2024-04-24T23:50:10.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.171+0000 7fd89ffff640 1 -- 172.21.15.16:0/4055266698 learned_addr learned my addr 172.21.15.16:0/4055266698 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:10.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.171+0000 7fd8a5442640 1 --2- 172.21.15.16:0/4055266698 >> v2:172.21.15.156:3300/0 conn(0x7fd8a00ced10 0x7fd8a006c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:10.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.172+0000 7fd8a4c41640 1 -- 172.21.15.16:0/4055266698 >> v2:172.21.15.156:3300/0 conn(0x7fd8a00ced10 msgr2=0x7fd8a006c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:10.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.172+0000 7fd8a4c41640 1 --2- 172.21.15.16:0/4055266698 >> v2:172.21.15.156:3300/0 conn(0x7fd8a00ced10 0x7fd8a006c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:10.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.172+0000 7fd8a4c41640 1 -- 172.21.15.16:0/4055266698 >> v1:172.21.15.16:6789/0 conn(0x7fd8a00b3060 legacy=0x7fd8a00ce600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:10.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.172+0000 7fd89f7fe640 1 -- 172.21.15.16:0/4055266698 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1137058391 0 0) 0x7fd8a00b3590 con 0x7fd8a00b3060 2024-04-24T23:50:10.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.172+0000 7fd8a4c41640 1 -- 172.21.15.16:0/4055266698 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a00b3b50 con 0x7fd8a01439d0 2024-04-24T23:50:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.172+0000 7fd8a4c41640 1 --2- 172.21.15.16:0/4055266698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8a01439d0 0x7fd8a00cf3d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00b4660 tx=0x7fd89802a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b48408e1beb8fdaf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.173+0000 7fd89f7fe640 1 -- 172.21.15.16:0/4055266698 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898007740 con 0x7fd8a01439d0 2024-04-24T23:50:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.173+0000 7fd89f7fe640 1 -- 172.21.15.16:0/4055266698 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8980078e0 con 0x7fd8a01439d0 2024-04-24T23:50:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.173+0000 7fd89f7fe640 1 -- 172.21.15.16:0/4055266698 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd898007cb0 con 0x7fd8a01439d0 2024-04-24T23:50:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.173+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/4055266698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8a01439d0 msgr2=0x7fd8a00cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:10.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.173+0000 7fd8a6ea5640 1 --2- 172.21.15.16:0/4055266698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8a01439d0 0x7fd8a00cf3d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00b4660 tx=0x7fd89802a680 comp rx=0 tx=0).stop 2024-04-24T23:50:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.173+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/4055266698 shutdown_connections 2024-04-24T23:50:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.173+0000 7fd8a6ea5640 1 --2- 172.21.15.16:0/4055266698 >> v2:172.21.15.156:3300/0 conn(0x7fd8a00ced10 0x7fd8a006c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.173+0000 7fd8a6ea5640 1 --2- 172.21.15.16:0/4055266698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8a01439d0 0x7fd8a00cf3d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.173+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/4055266698 >> 172.21.15.16:0/4055266698 conn(0x7fd8a00cfa20 msgr2=0x7fd8a00cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.173+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/4055266698 shutdown_connections 2024-04-24T23:50:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.173+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/4055266698 wait complete. 2024-04-24T23:50:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.174+0000 7fd8a6ea5640 1 Processor -- start 2024-04-24T23:50:10.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.174+0000 7fd8a6ea5640 1 -- start start 2024-04-24T23:50:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.175+0000 7fd8a6ea5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8a00b3060 0x7fd8a015a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:10.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.175+0000 7fd8a6ea5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8a00ced10 0x7fd8a0136d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.175+0000 7fd8a6ea5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8a01439d0 0x7fd8a01372a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.175+0000 7fd8a6ea5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a0144780 con 0x7fd8a01439d0 2024-04-24T23:50:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.175+0000 7fd8a6ea5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd8a00b4110 con 0x7fd8a00b3060 2024-04-24T23:50:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.175+0000 7fd8a6ea5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fd8a006c940 con 0x7fd8a00ced10 2024-04-24T23:50:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.175+0000 7fd8a5442640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8a01439d0 0x7fd8a01372a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.175+0000 7fd8a5442640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8a01439d0 0x7fd8a01372a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50612/0 (socket says 172.21.15.16:50612) 2024-04-24T23:50:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.175+0000 7fd8a5442640 1 -- 172.21.15.16:0/3521532287 learned_addr learned my addr 172.21.15.16:0/3521532287 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.175+0000 7fd8a4c41640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8a00b3060 0x7fd8a015a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.176+0000 7fd89ffff640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8a00ced10 0x7fd8a0136d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.176+0000 7fd8a5442640 1 -- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8a00ced10 msgr2=0x7fd8a0136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.176+0000 7fd8a5442640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8a00ced10 0x7fd8a0136d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.176+0000 7fd8a5442640 1 -- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8a00b3060 msgr2=0x7fd8a015a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:10.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.176+0000 7fd8a5442640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8a00b3060 0x7fd8a015a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.176+0000 7fd8a5442640 1 -- 172.21.15.16:0/3521532287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8a01399c0 con 0x7fd8a01439d0 2024-04-24T23:50:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.176+0000 7fd89ffff640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8a00ced10 0x7fd8a0136d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.176+0000 7fd8a5442640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8a01439d0 0x7fd8a01372a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd89000e930 tx=0x7fd89000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.176+0000 7fd89d7fa640 1 -- 172.21.15.16:0/3521532287 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd890017070 con 0x7fd8a01439d0 2024-04-24T23:50:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.176+0000 7fd89d7fa640 1 -- 172.21.15.16:0/3521532287 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd89000c980 con 0x7fd8a01439d0 2024-04-24T23:50:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.176+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/3521532287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8a0137ac0 con 0x7fd8a01439d0 2024-04-24T23:50:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.176+0000 7fd89d7fa640 1 -- 172.21.15.16:0/3521532287 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89000cd70 con 0x7fd8a01439d0 2024-04-24T23:50:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.176+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/3521532287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8a0138080 con 0x7fd8a01439d0 2024-04-24T23:50:10.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.177+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/3521532287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8a0139bc0 con 0x7fd8a01439d0 2024-04-24T23:50:10.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.178+0000 7fd89d7fa640 1 -- 172.21.15.16:0/3521532287 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd89001a070 con 0x7fd8a01439d0 2024-04-24T23:50:10.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.178+0000 7fd89d7fa640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd88006be30 0x7fd88006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:10.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.178+0000 7fd89d7fa640 1 -- 172.21.15.16:0/3521532287 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 6383+0+0 (secure 0 0 0) 0x7fd89008b0f0 con 0x7fd8a01439d0 2024-04-24T23:50:10.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.178+0000 7fd8a4c41640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd88006be30 0x7fd88006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:10.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.179+0000 7fd8a4c41640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd88006be30 0x7fd88006e2f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd898034000 tx=0x7fd89802f000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:10.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.182+0000 7fd89d7fa640 1 -- 172.21.15.16:0/3521532287 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd8a0139bc0 con 0x7fd8a01439d0 2024-04-24T23:50:10.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:10.367+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/3521532287 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd8a0068f90 con 0x7fd8a01439d0 2024-04-24T23:50:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.039+0000 7fd89d7fa640 1 -- 172.21.15.16:0/3521532287 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) ==== 169+0+0 (secure 0 0 0) 0x7fd8900591e0 con 0x7fd8a01439d0 2024-04-24T23:50:11.039 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-04-24T23:50:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.043+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd88006be30 msgr2=0x7fd88006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.043+0000 7fd8a6ea5640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd88006be30 0x7fd88006e2f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd898034000 tx=0x7fd89802f000 comp rx=0 tx=0).stop 2024-04-24T23:50:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.043+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/3521532287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8a01439d0 msgr2=0x7fd8a01372a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.043+0000 7fd8a6ea5640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8a01439d0 0x7fd8a01372a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fd89000e930 tx=0x7fd89000edf0 comp rx=0 tx=0).stop 2024-04-24T23:50:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.044+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/3521532287 shutdown_connections 2024-04-24T23:50:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.044+0000 7fd8a6ea5640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd88006be30 0x7fd88006e2f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.044+0000 7fd8a6ea5640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd8a01439d0 0x7fd8a01372a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.044+0000 7fd8a6ea5640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd8a00ced10 0x7fd8a0136d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.044+0000 7fd8a6ea5640 1 --2- 172.21.15.16:0/3521532287 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd8a00b3060 0x7fd8a015a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.044+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/3521532287 >> 172.21.15.16:0/3521532287 conn(0x7fd8a00cfa20 msgr2=0x7fd8a006d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:11.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.046+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/3521532287 shutdown_connections 2024-04-24T23:50:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.046+0000 7fd8a6ea5640 1 -- 172.21.15.16:0/3521532287 wait complete. 2024-04-24T23:50:11.063 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-24T23:50:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.179+0000 7f5e94c88640 1 Processor -- start 2024-04-24T23:50:11.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.179+0000 7f5e94c88640 1 -- start start 2024-04-24T23:50:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.180+0000 7f5e94c88640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5e900b5160 0x7f5e900711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.180+0000 7f5e94c88640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e900cc670 con 0x7f5e9015e2b0 2024-04-24T23:50:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.180+0000 7f5e94c88640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e900cdad0 con 0x7f5e901627a0 2024-04-24T23:50:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.180+0000 7f5e94c88640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f5e9015e690 con 0x7f5e900b5160 2024-04-24T23:50:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.180+0000 7f5e8e59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5e9015e2b0 0x7f5e900cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59024/0 (socket says 172.21.15.16:59024) 2024-04-24T23:50:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.180+0000 7f5e8e59c640 1 -- 172.21.15.16:0/2315260760 learned_addr learned my addr 172.21.15.16:0/2315260760 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.180+0000 7f5e8ed9d640 1 --2- 172.21.15.16:0/2315260760 >> v2:172.21.15.156:3300/0 conn(0x7f5e900b5160 0x7f5e900711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.180+0000 7f5e8d59a640 1 -- 172.21.15.16:0/2315260760 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2146570830 0 0) 0x7f5e900cc670 con 0x7f5e9015e2b0 2024-04-24T23:50:11.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.180+0000 7f5e8d59a640 1 -- 172.21.15.16:0/2315260760 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e78003520 con 0x7f5e9015e2b0 2024-04-24T23:50:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8d59a640 1 -- 172.21.15.16:0/2315260760 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3376899613 0 0) 0x7f5e78003520 con 0x7f5e9015e2b0 2024-04-24T23:50:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8d59a640 1 -- 172.21.15.16:0/2315260760 >> v2:172.21.15.156:3300/0 conn(0x7f5e900b5160 msgr2=0x7f5e900711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8d59a640 1 --2- 172.21.15.16:0/2315260760 >> v2:172.21.15.156:3300/0 conn(0x7f5e900b5160 0x7f5e900711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8d59a640 1 -- 172.21.15.16:0/2315260760 >> v1:172.21.15.156:6789/0 conn(0x7f5e901627a0 legacy=0x7f5e900b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8d59a640 1 -- 172.21.15.16:0/2315260760 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e900cc8e0 con 0x7f5e9015e2b0 2024-04-24T23:50:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8d59a640 1 -- 172.21.15.16:0/2315260760 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f5e80001670 con 0x7f5e9015e2b0 2024-04-24T23:50:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8d59a640 1 -- 172.21.15.16:0/2315260760 >> v1:172.21.15.16:6789/0 conn(0x7f5e9015e2b0 legacy=0x7f5e900cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8d59a640 1 --2- 172.21.15.16:0/2315260760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e780039b0 0x7f5e78003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8d59a640 1 --2- 172.21.15.16:0/2315260760 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e780043b0 0x7f5e78004880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8d59a640 1 --2- 172.21.15.16:0/2315260760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e78005080 0x7f5e7800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8d59a640 1 -- 172.21.15.16:0/2315260760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5e9015e690 con 0x7f5e780039b0 2024-04-24T23:50:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8d59a640 1 -- 172.21.15.16:0/2315260760 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5e7800bfe0 con 0x7f5e78005080 2024-04-24T23:50:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8d59a640 1 -- 172.21.15.16:0/2315260760 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f5e7800c1e0 con 0x7f5e780043b0 2024-04-24T23:50:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8e59c640 1 --2- 172.21.15.16:0/2315260760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e780039b0 0x7f5e78003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8ed9d640 1 --2- 172.21.15.16:0/2315260760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e78005080 0x7f5e7800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8dd9b640 1 --2- 172.21.15.16:0/2315260760 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e780043b0 0x7f5e78004880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8e59c640 1 -- 172.21.15.16:0/2315260760 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e780043b0 msgr2=0x7f5e78004880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8e59c640 1 --2- 172.21.15.16:0/2315260760 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e780043b0 0x7f5e78004880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8e59c640 1 -- 172.21.15.16:0/2315260760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e78005080 msgr2=0x7f5e7800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8e59c640 1 --2- 172.21.15.16:0/2315260760 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e78005080 0x7f5e7800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8ed9d640 1 -- 172.21.15.16:0/2315260760 reap_dead start 2024-04-24T23:50:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.181+0000 7f5e8e59c640 1 -- 172.21.15.16:0/2315260760 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e78003520 con 0x7f5e780039b0 2024-04-24T23:50:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.182+0000 7f5e8e59c640 1 --2- 172.21.15.16:0/2315260760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e780039b0 0x7f5e78003d90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000c190 tx=0x7f5e78005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4027f529fe0262e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.182+0000 7f5e8d59a640 1 -- 172.21.15.16:0/2315260760 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e80007cc0 con 0x7f5e780039b0 2024-04-24T23:50:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.182+0000 7f5e8d59a640 1 -- 172.21.15.16:0/2315260760 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e80005ad0 con 0x7f5e780039b0 2024-04-24T23:50:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.182+0000 7f5e8d59a640 1 -- 172.21.15.16:0/2315260760 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e80007070 con 0x7f5e780039b0 2024-04-24T23:50:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.183+0000 7f5e94c88640 1 -- 172.21.15.16:0/2315260760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e780039b0 msgr2=0x7f5e78003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:11.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.183+0000 7f5e94c88640 1 --2- 172.21.15.16:0/2315260760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e780039b0 0x7f5e78003d90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000c190 tx=0x7f5e78005830 comp rx=0 tx=0).stop 2024-04-24T23:50:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.183+0000 7f5e94c88640 1 -- 172.21.15.16:0/2315260760 shutdown_connections 2024-04-24T23:50:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.183+0000 7f5e94c88640 1 --2- 172.21.15.16:0/2315260760 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e780039b0 0x7f5e78003d90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.183+0000 7f5e94c88640 1 -- 172.21.15.16:0/2315260760 >> 172.21.15.16:0/2315260760 conn(0x7f5e900d1bc0 msgr2=0x7f5e900d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.183+0000 7f5e94c88640 1 -- 172.21.15.16:0/2315260760 shutdown_connections 2024-04-24T23:50:11.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.183+0000 7f5e94c88640 1 -- 172.21.15.16:0/2315260760 wait complete. 2024-04-24T23:50:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.184+0000 7f5e94c88640 1 Processor -- start 2024-04-24T23:50:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.184+0000 7f5e94c88640 1 -- start start 2024-04-24T23:50:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e94c88640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e780039b0 0x7f5e9015c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e94c88640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e9015cf70 0x7f5e900afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e94c88640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e900c56f0 0x7f5e900c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e94c88640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5e90162c30 con 0x7f5e9015cf70 2024-04-24T23:50:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e94c88640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5e900ccf30 con 0x7f5e780039b0 2024-04-24T23:50:11.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e94c88640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f5e900cdf30 con 0x7f5e900c56f0 2024-04-24T23:50:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e8dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e9015cf70 0x7f5e900afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e8dd9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e9015cf70 0x7f5e900afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50628/0 (socket says 172.21.15.16:50628) 2024-04-24T23:50:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e8dd9b640 1 -- 172.21.15.16:0/410679767 learned_addr learned my addr 172.21.15.16:0/410679767 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e8e59c640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e780039b0 0x7f5e9015c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e8dd9b640 1 -- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e900c56f0 msgr2=0x7f5e900c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e8ed9d640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e900c56f0 0x7f5e900c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e8dd9b640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e900c56f0 0x7f5e900c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e8dd9b640 1 -- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e780039b0 msgr2=0x7f5e9015c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e8dd9b640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e780039b0 0x7f5e9015c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e8dd9b640 1 -- 172.21.15.16:0/410679767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e900aeae0 con 0x7f5e9015cf70 2024-04-24T23:50:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e8e59c640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e780039b0 0x7f5e9015c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.185+0000 7f5e8ed9d640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e900c56f0 0x7f5e900c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:11.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.186+0000 7f5e8dd9b640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e9015cf70 0x7f5e900afb80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5e8400f8d0 tx=0x7f5e8400fd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.186+0000 7f5e7effd640 1 -- 172.21.15.16:0/410679767 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e8401a070 con 0x7f5e9015cf70 2024-04-24T23:50:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.186+0000 7f5e7effd640 1 -- 172.21.15.16:0/410679767 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e84005320 con 0x7f5e9015cf70 2024-04-24T23:50:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.186+0000 7f5e7effd640 1 -- 172.21.15.16:0/410679767 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e84007270 con 0x7f5e9015cf70 2024-04-24T23:50:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.186+0000 7f5e94c88640 1 -- 172.21.15.16:0/410679767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e900aedc0 con 0x7f5e9015cf70 2024-04-24T23:50:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.186+0000 7f5e94c88640 1 -- 172.21.15.16:0/410679767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e900af380 con 0x7f5e9015cf70 2024-04-24T23:50:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.187+0000 7f5e94c88640 1 -- 172.21.15.16:0/410679767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e900b4660 con 0x7f5e9015cf70 2024-04-24T23:50:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.187+0000 7f5e7effd640 1 -- 172.21.15.16:0/410679767 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5e8401d070 con 0x7f5e9015cf70 2024-04-24T23:50:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.188+0000 7f5e7effd640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5e7006be30 0x7f5e7006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:11.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.188+0000 7f5e7effd640 1 -- 172.21.15.16:0/410679767 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5853+0+0 (secure 0 0 0) 0x7f5e8408a7c0 con 0x7f5e9015cf70 2024-04-24T23:50:11.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.188+0000 7f5e8e59c640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5e7006be30 0x7f5e7006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:11.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.188+0000 7f5e8e59c640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5e7006be30 0x7f5e7006e2f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5e80032000 tx=0x7f5e8002d000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:11.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.192+0000 7f5e7effd640 1 -- 172.21.15.16:0/410679767 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5e900b4660 con 0x7f5e9015cf70 2024-04-24T23:50:11.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:11.374+0000 7f5e94c88640 1 -- 172.21.15.16:0/410679767 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f5e90068f90 con 0x7f5e9015cf70 2024-04-24T23:50:12.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.044+0000 7f5e7effd640 1 -- 172.21.15.16:0/410679767 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) ==== 178+0+0 (secure 0 0 0) 0x7f5e840590e0 con 0x7f5e9015cf70 2024-04-24T23:50:12.044 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data_ec' removed 2024-04-24T23:50:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.050+0000 7f5e94c88640 1 -- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5e7006be30 msgr2=0x7f5e7006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.050+0000 7f5e94c88640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5e7006be30 0x7f5e7006e2f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5e80032000 tx=0x7f5e8002d000 comp rx=0 tx=0).stop 2024-04-24T23:50:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.050+0000 7f5e94c88640 1 -- 172.21.15.16:0/410679767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e9015cf70 msgr2=0x7f5e900afb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.050+0000 7f5e94c88640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e9015cf70 0x7f5e900afb80 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5e8400f8d0 tx=0x7f5e8400fd90 comp rx=0 tx=0).stop 2024-04-24T23:50:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.050+0000 7f5e94c88640 1 -- 172.21.15.16:0/410679767 shutdown_connections 2024-04-24T23:50:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.050+0000 7f5e94c88640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5e7006be30 0x7f5e7006e2f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.050+0000 7f5e94c88640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5e900c56f0 0x7f5e900c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.050+0000 7f5e94c88640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5e9015cf70 0x7f5e900afb80 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.050+0000 7f5e94c88640 1 --2- 172.21.15.16:0/410679767 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5e780039b0 0x7f5e9015c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.050+0000 7f5e94c88640 1 -- 172.21.15.16:0/410679767 >> 172.21.15.16:0/410679767 conn(0x7f5e900d1bc0 msgr2=0x7f5e900cd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.051+0000 7f5e94c88640 1 -- 172.21.15.16:0/410679767 shutdown_connections 2024-04-24T23:50:12.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.051+0000 7f5e94c88640 1 -- 172.21.15.16:0/410679767 wait complete. 2024-04-24T23:50:12.068 INFO:tasks.ceph.mds.a:Restarting daemon 2024-04-24T23:50:12.068 INFO:tasks.ceph.mds.a:Stopping old one... 2024-04-24T23:50:12.068 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-24T23:50:12.068 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:50:12.069 INFO:tasks.ceph.mds.c:Restarting daemon 2024-04-24T23:50:12.069 INFO:tasks.ceph.mds.c:Stopping old one... 2024-04-24T23:50:12.069 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-24T23:50:12.069 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:50:12.070 INFO:tasks.ceph.mds.b:Restarting daemon 2024-04-24T23:50:12.070 INFO:tasks.ceph.mds.b:Stopping old one... 2024-04-24T23:50:12.070 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-24T23:50:12.070 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:50:12.071 INFO:tasks.ceph.mds.d:Restarting daemon 2024-04-24T23:50:12.071 INFO:tasks.ceph.mds.d:Stopping old one... 2024-04-24T23:50:12.071 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-24T23:50:12.071 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:50:12.079 INFO:tasks.ceph.mds.a:Stopped 2024-04-24T23:50:12.079 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-04-24T23:50:12.080 INFO:tasks.ceph.mds.c:Stopped 2024-04-24T23:50:12.080 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-04-24T23:50:12.082 INFO:tasks.ceph.mds.b:Stopped 2024-04-24T23:50:12.083 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-04-24T23:50:12.083 INFO:tasks.ceph.mds.d:Stopped 2024-04-24T23:50:12.083 DEBUG:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-04-24T23:50:12.123 INFO:tasks.ceph.mds.a:Started 2024-04-24T23:50:12.123 DEBUG:teuthology.parallel:result is None 2024-04-24T23:50:12.124 INFO:tasks.ceph.mds.c:Started 2024-04-24T23:50:12.124 DEBUG:teuthology.parallel:result is None 2024-04-24T23:50:12.127 INFO:tasks.ceph.mds.b:Started 2024-04-24T23:50:12.127 DEBUG:teuthology.parallel:result is None 2024-04-24T23:50:12.128 INFO:tasks.ceph.mds.d:Started 2024-04-24T23:50:12.128 DEBUG:teuthology.parallel:result is None 2024-04-24T23:50:12.128 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-04-24T23:50:12.203 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-04-24T23:50:12.204 INFO:tasks.ceph.mds.c.smithi016.stdout:starting mds.c at 2024-04-24T23:50:12.209 INFO:tasks.ceph.mds.b.smithi156.stdout:starting mds.b at 2024-04-24T23:50:12.210 INFO:tasks.ceph.mds.d.smithi156.stdout:starting mds.d at 2024-04-24T23:50:12.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.243+0000 7f5d9e800640 1 Processor -- start 2024-04-24T23:50:12.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.243+0000 7f5d9e800640 1 -- start start 2024-04-24T23:50:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.244+0000 7f5d9e800640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d9815df20 0x7f5d98071a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.244+0000 7f5d9e800640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f5d980d0530 0x7f5d98071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.244+0000 7f5d9e800640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d980b6a70 con 0x7f5d980cfe50 2024-04-24T23:50:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.244+0000 7f5d9e800640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f5d980cf460 con 0x7f5d9815df20 2024-04-24T23:50:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.244+0000 7f5d9e800640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f5d980b6820 con 0x7f5d980d0530 2024-04-24T23:50:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.244+0000 7f5d97fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d9815df20 0x7f5d98071a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.244+0000 7f5d97fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d9815df20 0x7f5d98071a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50674/0 (socket says 172.21.15.16:50674) 2024-04-24T23:50:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.244+0000 7f5d97fff640 1 -- 172.21.15.16:0/1974708624 learned_addr learned my addr 172.21.15.16:0/1974708624 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.244+0000 7f5d977fe640 1 --2- 172.21.15.16:0/1974708624 >> v2:172.21.15.156:3300/0 conn(0x7f5d980d0530 0x7f5d98071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.245+0000 7f5d97fff640 1 -- 172.21.15.16:0/1974708624 >> v2:172.21.15.156:3300/0 conn(0x7f5d980d0530 msgr2=0x7f5d98071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.245+0000 7f5d97fff640 1 --2- 172.21.15.16:0/1974708624 >> v2:172.21.15.156:3300/0 conn(0x7f5d980d0530 0x7f5d98071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.245+0000 7f5d97fff640 1 -- 172.21.15.16:0/1974708624 >> v1:172.21.15.156:6789/0 conn(0x7f5d980cfe50 legacy=0x7f5d980ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.245+0000 7f5d97fff640 1 -- 172.21.15.16:0/1974708624 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d980ce6a0 con 0x7f5d9815df20 2024-04-24T23:50:12.255 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.245+0000 7f5d97fff640 1 --2- 172.21.15.16:0/1974708624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d9815df20 0x7f5d98071a00 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5d981619c0 tx=0x7f5d8802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2db0cc12aff92 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.246+0000 7f5d96ffd640 1 -- 172.21.15.16:0/1974708624 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d88037070 con 0x7f5d9815df20 2024-04-24T23:50:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.246+0000 7f5d96ffd640 1 -- 172.21.15.16:0/1974708624 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d88032440 con 0x7f5d9815df20 2024-04-24T23:50:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.246+0000 7f5d96ffd640 1 -- 172.21.15.16:0/1974708624 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d88032810 con 0x7f5d9815df20 2024-04-24T23:50:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.246+0000 7f5d9e800640 1 -- 172.21.15.16:0/1974708624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d9815df20 msgr2=0x7f5d98071a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.246+0000 7f5d9e800640 1 --2- 172.21.15.16:0/1974708624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d9815df20 0x7f5d98071a00 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f5d981619c0 tx=0x7f5d8802a270 comp rx=0 tx=0).stop 2024-04-24T23:50:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.246+0000 7f5d9e800640 1 -- 172.21.15.16:0/1974708624 shutdown_connections 2024-04-24T23:50:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.246+0000 7f5d9e800640 1 --2- 172.21.15.16:0/1974708624 >> v2:172.21.15.156:3300/0 conn(0x7f5d980d0530 0x7f5d98071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.246+0000 7f5d9e800640 1 --2- 172.21.15.16:0/1974708624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d9815df20 0x7f5d98071a00 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.246+0000 7f5d9e800640 1 -- 172.21.15.16:0/1974708624 >> 172.21.15.16:0/1974708624 conn(0x7f5d980d0ac0 msgr2=0x7f5d980d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.246+0000 7f5d9e800640 1 -- 172.21.15.16:0/1974708624 shutdown_connections 2024-04-24T23:50:12.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.246+0000 7f5d9e800640 1 -- 172.21.15.16:0/1974708624 wait complete. 2024-04-24T23:50:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.247+0000 7f5d9e800640 1 Processor -- start 2024-04-24T23:50:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.248+0000 7f5d9e800640 1 -- start start 2024-04-24T23:50:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.248+0000 7f5d9e800640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d980cfe50 0x7f5d980adb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.248+0000 7f5d9e800640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d980d0530 0x7f5d9814f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.248+0000 7f5d9e800640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d9815df20 0x7f5d9814fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.248+0000 7f5d9e800640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5d980cef40 con 0x7f5d9815df20 2024-04-24T23:50:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.248+0000 7f5d9e800640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5d980cec60 con 0x7f5d980cfe50 2024-04-24T23:50:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.248+0000 7f5d9e800640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f5d980b4030 con 0x7f5d980d0530 2024-04-24T23:50:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d9cd9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d9815df20 0x7f5d9814fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d9cd9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d9815df20 0x7f5d9814fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50684/0 (socket says 172.21.15.16:50684) 2024-04-24T23:50:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d9cd9d640 1 -- 172.21.15.16:0/3746824021 learned_addr learned my addr 172.21.15.16:0/3746824021 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d97fff640 1 --2- 172.21.15.16:0/3746824021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d980cfe50 0x7f5d980adb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d977fe640 1 --2- 172.21.15.16:0/3746824021 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d980d0530 0x7f5d9814f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d9cd9d640 1 -- 172.21.15.16:0/3746824021 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d980d0530 msgr2=0x7f5d9814f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d9cd9d640 1 --2- 172.21.15.16:0/3746824021 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d980d0530 0x7f5d9814f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d9cd9d640 1 -- 172.21.15.16:0/3746824021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d980cfe50 msgr2=0x7f5d980adb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d9cd9d640 1 --2- 172.21.15.16:0/3746824021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d980cfe50 0x7f5d980adb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d9cd9d640 1 -- 172.21.15.16:0/3746824021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d9806baf0 con 0x7f5d9815df20 2024-04-24T23:50:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d9cd9d640 1 --2- 172.21.15.16:0/3746824021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d9815df20 0x7f5d9814fdf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5d9000c730 tx=0x7f5d9000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d94ff9640 1 -- 172.21.15.16:0/3746824021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d90003b70 con 0x7f5d9815df20 2024-04-24T23:50:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d94ff9640 1 -- 172.21.15.16:0/3746824021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d90003d10 con 0x7f5d9815df20 2024-04-24T23:50:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d9e800640 1 -- 172.21.15.16:0/3746824021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d9806bdd0 con 0x7f5d9815df20 2024-04-24T23:50:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d94ff9640 1 -- 172.21.15.16:0/3746824021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d9000aa90 con 0x7f5d9815df20 2024-04-24T23:50:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.249+0000 7f5d9e800640 1 -- 172.21.15.16:0/3746824021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d9806c390 con 0x7f5d9815df20 2024-04-24T23:50:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.250+0000 7f5d9e800640 1 -- 172.21.15.16:0/3746824021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d9806c590 con 0x7f5d9815df20 2024-04-24T23:50:12.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.251+0000 7f5d94ff9640 1 -- 172.21.15.16:0/3746824021 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5d9000ac30 con 0x7f5d9815df20 2024-04-24T23:50:12.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.251+0000 7f5d94ff9640 1 --2- 172.21.15.16:0/3746824021 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5d8406be30 0x7f5d8406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:12.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.251+0000 7f5d94ff9640 1 -- 172.21.15.16:0/3746824021 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5335+0+0 (secure 0 0 0) 0x7f5d9008ab40 con 0x7f5d9815df20 2024-04-24T23:50:12.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.251+0000 7f5d97fff640 1 --2- 172.21.15.16:0/3746824021 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5d8406be30 0x7f5d8406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:12.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.252+0000 7f5d97fff640 1 --2- 172.21.15.16:0/3746824021 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5d8406be30 0x7f5d8406e2f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5d88034000 tx=0x7f5d8802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:12.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.255+0000 7f5d94ff9640 1 -- 172.21.15.16:0/3746824021 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5d9806c590 con 0x7f5d9815df20 2024-04-24T23:50:12.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:12.449+0000 7f5d9e800640 1 -- 172.21.15.16:0/3746824021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f5d9806f8f0 con 0x7f5d9815df20 2024-04-24T23:50:13.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.052+0000 7f5d94ff9640 1 -- 172.21.15.16:0/3746824021 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v33) ==== 97+0+0 (secure 0 0 0) 0x7f5d90058f90 con 0x7f5d9815df20 2024-04-24T23:50:13.052 INFO:teuthology.orchestra.run.smithi016.stderr: removed all blocklist entries 2024-04-24T23:50:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.056+0000 7f5d9e800640 1 -- 172.21.15.16:0/3746824021 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5d8406be30 msgr2=0x7f5d8406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.056+0000 7f5d9e800640 1 --2- 172.21.15.16:0/3746824021 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5d8406be30 0x7f5d8406e2f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f5d88034000 tx=0x7f5d8802e000 comp rx=0 tx=0).stop 2024-04-24T23:50:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.056+0000 7f5d9e800640 1 -- 172.21.15.16:0/3746824021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d9815df20 msgr2=0x7f5d9814fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.056+0000 7f5d9e800640 1 --2- 172.21.15.16:0/3746824021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d9815df20 0x7f5d9814fdf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5d9000c730 tx=0x7f5d9000cbf0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.056+0000 7f5d9e800640 1 -- 172.21.15.16:0/3746824021 shutdown_connections 2024-04-24T23:50:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.056+0000 7f5d9e800640 1 --2- 172.21.15.16:0/3746824021 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5d8406be30 0x7f5d8406e2f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.056+0000 7f5d9e800640 1 --2- 172.21.15.16:0/3746824021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d9815df20 0x7f5d9814fdf0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.056+0000 7f5d9e800640 1 --2- 172.21.15.16:0/3746824021 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d980d0530 0x7f5d9814f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.056+0000 7f5d9e800640 1 --2- 172.21.15.16:0/3746824021 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d980cfe50 0x7f5d980adb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.056+0000 7f5d9e800640 1 -- 172.21.15.16:0/3746824021 >> 172.21.15.16:0/3746824021 conn(0x7f5d980d0ac0 msgr2=0x7f5d980d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:13.056 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.056+0000 7f5d9e800640 1 -- 172.21.15.16:0/3746824021 shutdown_connections 2024-04-24T23:50:13.057 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.056+0000 7f5d9e800640 1 -- 172.21.15.16:0/3746824021 wait complete. 2024-04-24T23:50:13.071 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-04-24T23:50:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.177+0000 7f37b67ce640 1 Processor -- start 2024-04-24T23:50:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.178+0000 7f37b67ce640 1 -- start start 2024-04-24T23:50:13.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.178+0000 7f37b67ce640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f37b00729a0 0x7f37b0161c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.178+0000 7f37b67ce640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f37b01621b0 0x7f37b00b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.178+0000 7f37b67ce640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37b0162a10 con 0x7f37b015df40 2024-04-24T23:50:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.178+0000 7f37b67ce640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f37b00ae2b0 con 0x7f37b01621b0 2024-04-24T23:50:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.178+0000 7f37b67ce640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f37b00ad980 con 0x7f37b00729a0 2024-04-24T23:50:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.178+0000 7f37b4d6b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f37b01621b0 0x7f37b00b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.179+0000 7f37b4d6b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f37b01621b0 0x7f37b00b3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50688/0 (socket says 172.21.15.16:50688) 2024-04-24T23:50:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.179+0000 7f37b4d6b640 1 -- 172.21.15.16:0/1898694409 learned_addr learned my addr 172.21.15.16:0/1898694409 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.179+0000 7f37af7fe640 1 --2- 172.21.15.16:0/1898694409 >> v2:172.21.15.156:3300/0 conn(0x7f37b00729a0 0x7f37b0161c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.179+0000 7f37b4d6b640 1 -- 172.21.15.16:0/1898694409 >> v2:172.21.15.156:3300/0 conn(0x7f37b00729a0 msgr2=0x7f37b0161c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.179+0000 7f37b4d6b640 1 --2- 172.21.15.16:0/1898694409 >> v2:172.21.15.156:3300/0 conn(0x7f37b00729a0 0x7f37b0161c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.179+0000 7f37b4d6b640 1 -- 172.21.15.16:0/1898694409 >> v1:172.21.15.156:6789/0 conn(0x7f37b015df40 legacy=0x7f37b0072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.179+0000 7f37b4d6b640 1 -- 172.21.15.16:0/1898694409 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b00adc60 con 0x7f37b01621b0 2024-04-24T23:50:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.179+0000 7f37af7fe640 1 --2- 172.21.15.16:0/1898694409 >> v2:172.21.15.156:3300/0 conn(0x7f37b00729a0 0x7f37b0161c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.179+0000 7f37b4d6b640 1 --2- 172.21.15.16:0/1898694409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37b01621b0 0x7f37b00b3bf0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f37b0071210 tx=0x7f37a802a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94663495b2765920 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.180+0000 7f37aeffd640 1 -- 172.21.15.16:0/1898694409 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a8036070 con 0x7f37b01621b0 2024-04-24T23:50:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.180+0000 7f37aeffd640 1 -- 172.21.15.16:0/1898694409 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37a8031600 con 0x7f37b01621b0 2024-04-24T23:50:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.180+0000 7f37aeffd640 1 -- 172.21.15.16:0/1898694409 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a80319d0 con 0x7f37b01621b0 2024-04-24T23:50:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.180+0000 7f37b67ce640 1 -- 172.21.15.16:0/1898694409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37b01621b0 msgr2=0x7f37b00b3bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.180+0000 7f37b67ce640 1 --2- 172.21.15.16:0/1898694409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37b01621b0 0x7f37b00b3bf0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f37b0071210 tx=0x7f37a802a6f0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.180+0000 7f37b67ce640 1 -- 172.21.15.16:0/1898694409 shutdown_connections 2024-04-24T23:50:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.180+0000 7f37b67ce640 1 --2- 172.21.15.16:0/1898694409 >> v2:172.21.15.156:3300/0 conn(0x7f37b00729a0 0x7f37b0161c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.180+0000 7f37b67ce640 1 --2- 172.21.15.16:0/1898694409 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37b01621b0 0x7f37b00b3bf0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.180+0000 7f37b67ce640 1 -- 172.21.15.16:0/1898694409 >> 172.21.15.16:0/1898694409 conn(0x7f37b00d0260 msgr2=0x7f37b00d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.180+0000 7f37b67ce640 1 -- 172.21.15.16:0/1898694409 shutdown_connections 2024-04-24T23:50:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.180+0000 7f37b67ce640 1 -- 172.21.15.16:0/1898694409 wait complete. 2024-04-24T23:50:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.181+0000 7f37b67ce640 1 Processor -- start 2024-04-24T23:50:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.182+0000 7f37b67ce640 1 -- start start 2024-04-24T23:50:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.182+0000 7f37b67ce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37b00729a0 0x7f37b014ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.182+0000 7f37b67ce640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f37b015df40 0x7f37b0157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.182+0000 7f37b67ce640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37b01621b0 0x7f37b006d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.182+0000 7f37b67ce640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f37b015e320 con 0x7f37b00729a0 2024-04-24T23:50:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.182+0000 7f37b67ce640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f37b00ae740 con 0x7f37b01621b0 2024-04-24T23:50:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.182+0000 7f37b67ce640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f37b015e4a0 con 0x7f37b015df40 2024-04-24T23:50:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.182+0000 7f37affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37b00729a0 0x7f37b014ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.182+0000 7f37affff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37b00729a0 0x7f37b014ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50702/0 (socket says 172.21.15.16:50702) 2024-04-24T23:50:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.182+0000 7f37affff640 1 -- 172.21.15.16:0/704961693 learned_addr learned my addr 172.21.15.16:0/704961693 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.183+0000 7f37b4d6b640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37b01621b0 0x7f37b006d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.183+0000 7f37af7fe640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f37b015df40 0x7f37b0157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.183+0000 7f37affff640 1 -- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f37b015df40 msgr2=0x7f37b0157990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.183+0000 7f37affff640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f37b015df40 0x7f37b0157990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.183+0000 7f37affff640 1 -- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37b01621b0 msgr2=0x7f37b006d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.183+0000 7f37affff640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37b01621b0 0x7f37b006d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.183+0000 7f37affff640 1 -- 172.21.15.16:0/704961693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37b00b1000 con 0x7f37b00729a0 2024-04-24T23:50:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.183+0000 7f37b4d6b640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37b01621b0 0x7f37b006d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.183+0000 7f37affff640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37b00729a0 0x7f37b014ebd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f37a000c430 tx=0x7f37a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.183+0000 7f37acff9640 1 -- 172.21.15.16:0/704961693 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0011790 con 0x7f37b00729a0 2024-04-24T23:50:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.183+0000 7f37acff9640 1 -- 172.21.15.16:0/704961693 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37a0017070 con 0x7f37b00729a0 2024-04-24T23:50:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.183+0000 7f37b67ce640 1 -- 172.21.15.16:0/704961693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37b00b12e0 con 0x7f37b00729a0 2024-04-24T23:50:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.183+0000 7f37acff9640 1 -- 172.21.15.16:0/704961693 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37a0011bb0 con 0x7f37b00729a0 2024-04-24T23:50:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.183+0000 7f37b67ce640 1 -- 172.21.15.16:0/704961693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37b00b18a0 con 0x7f37b00729a0 2024-04-24T23:50:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.184+0000 7f37b67ce640 1 -- 172.21.15.16:0/704961693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37b006e5c0 con 0x7f37b00729a0 2024-04-24T23:50:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.185+0000 7f37acff9640 1 -- 172.21.15.16:0/704961693 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f37a001c460 con 0x7f37b00729a0 2024-04-24T23:50:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.185+0000 7f37acff9640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f379c06be30 0x7f379c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.185+0000 7f37acff9640 1 -- 172.21.15.16:0/704961693 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f37a008aba0 con 0x7f37b00729a0 2024-04-24T23:50:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.185+0000 7f37af7fe640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f379c06be30 0x7f379c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.186+0000 7f37af7fe640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f379c06be30 0x7f379c06e2f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f37b0072020 tx=0x7f37a4008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.189+0000 7f37acff9640 1 -- 172.21.15.16:0/704961693 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f37a0059710 con 0x7f37b00729a0 2024-04-24T23:50:13.439 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.440+0000 7f37b67ce640 1 -- 172.21.15.16:0/704961693 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f37b00d0040 con 0x7f37b00729a0 2024-04-24T23:50:13.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.440+0000 7f37acff9640 1 -- 172.21.15.16:0/704961693 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7f37a0059010 con 0x7f37b00729a0 2024-04-24T23:50:13.440 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:13.440 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:50:13.441 INFO:teuthology.orchestra.run.smithi016.stdout: "auth_dump": [ 2024-04-24T23:50:13.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.441 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.a", 2024-04-24T23:50:13.441 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAsmilmGXU4EBAAot69Ad+4q0ajnN525mELrA==", 2024-04-24T23:50:13.441 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.441 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-04-24T23:50:13.441 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-04-24T23:50:13.441 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-04-24T23:50:13.441 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.441 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.441 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.441 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.442 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.b", 2024-04-24T23:50:13.442 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAsmilmwriuGxAA8IeWbl5CQnyegSGw47x7Qg==", 2024-04-24T23:50:13.442 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.442 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-04-24T23:50:13.442 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-04-24T23:50:13.442 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-04-24T23:50:13.442 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.442 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.442 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.442 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.442 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.c", 2024-04-24T23:50:13.442 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAsmilmhlw3FxAAi8h6MHJf5CvKyTOCBTYbkA==", 2024-04-24T23:50:13.443 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.443 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-04-24T23:50:13.443 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-04-24T23:50:13.443 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-04-24T23:50:13.443 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.443 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.443 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.443 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.443 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mds.d", 2024-04-24T23:50:13.443 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAsmilm+Dt+IhAAPk+MsjISFV980KLFyT5LiA==", 2024-04-24T23:50:13.443 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.443 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-04-24T23:50:13.443 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-04-24T23:50:13.444 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-04-24T23:50:13.444 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.444 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.444 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.444 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.444 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.0", 2024-04-24T23:50:13.444 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAumilmBjEvCRAAGfXfrApvnEA4UsH2LLrOtw==", 2024-04-24T23:50:13.444 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.444 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-04-24T23:50:13.444 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-04-24T23:50:13.444 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.444 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.445 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.1", 2024-04-24T23:50:13.445 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAxmilmyHSaBxAAo8nFyeMCWXSoZsQ0BpeOnQ==", 2024-04-24T23:50:13.445 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.445 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-04-24T23:50:13.445 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-04-24T23:50:13.445 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.445 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.445 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.445 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.445 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.2", 2024-04-24T23:50:13.445 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQA0milmi8e/BRAAKHTiaAzRVBDzUbMETp2Z3Q==", 2024-04-24T23:50:13.446 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.446 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-04-24T23:50:13.446 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-04-24T23:50:13.446 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.446 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.446 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.446 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.446 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.3", 2024-04-24T23:50:13.446 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQA3milmnu29AxAA1obUjImx0vtxWob+Rg9tVA==", 2024-04-24T23:50:13.446 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.446 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-04-24T23:50:13.446 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-04-24T23:50:13.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.447 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.4", 2024-04-24T23:50:13.447 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQA7milmFHvpDRAAOIPyH67azej6WG7kLQR/Dw==", 2024-04-24T23:50:13.447 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.447 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-04-24T23:50:13.447 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-04-24T23:50:13.447 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.447 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.447 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.447 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.448 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.5", 2024-04-24T23:50:13.448 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQA+milmEWmcCxAAPY/2viwca5G+6T6JWIjkDQ==", 2024-04-24T23:50:13.448 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.448 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-04-24T23:50:13.448 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-04-24T23:50:13.448 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.448 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.448 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.448 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.448 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.6", 2024-04-24T23:50:13.448 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQBBmilmtjSoCRAAjtwouhdVLqMb6OrTuq1nUw==", 2024-04-24T23:50:13.448 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.449 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-04-24T23:50:13.449 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-04-24T23:50:13.449 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.449 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.449 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.449 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.449 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "osd.7", 2024-04-24T23:50:13.449 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQBEmilmDB1ICBAAH5A6jnsXpdpw46yRLIigCg==", 2024-04-24T23:50:13.449 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.449 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow profile osd", 2024-04-24T23:50:13.449 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile osd", 2024-04-24T23:50:13.449 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.449 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.450 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.0", 2024-04-24T23:50:13.450 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAsmilm81VpJhAAXQnTDv4ShV2nH4Tpv5oziw==", 2024-04-24T23:50:13.450 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.450 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-04-24T23:50:13.450 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-04-24T23:50:13.450 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-04-24T23:50:13.450 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-04-24T23:50:13.450 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.450 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.450 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.451 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.1", 2024-04-24T23:50:13.451 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAsmilmfVgCLxAAqmZ7wgN04zvzOcdv/Q6CVQ==", 2024-04-24T23:50:13.451 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.451 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-04-24T23:50:13.451 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-04-24T23:50:13.451 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-04-24T23:50:13.451 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-04-24T23:50:13.451 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.451 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.451 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.451 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.2", 2024-04-24T23:50:13.451 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAsmilmC+jxKxAAt1KAO9bdg0WndKIqvNJBCA==", 2024-04-24T23:50:13.452 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.452 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-04-24T23:50:13.452 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-04-24T23:50:13.452 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-04-24T23:50:13.452 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-04-24T23:50:13.452 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.452 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.452 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.452 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.3", 2024-04-24T23:50:13.452 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAsmilmpkNrNBAAfW/Xales/f5k4EgzHcqnkQ==", 2024-04-24T23:50:13.452 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.452 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow", 2024-04-24T23:50:13.452 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow r", 2024-04-24T23:50:13.453 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow rw", 2024-04-24T23:50:13.453 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow rwx" 2024-04-24T23:50:13.453 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.453 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.453 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.453 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.admin", 2024-04-24T23:50:13.453 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQArmilmS8b+MBAAT3hKsR6Xamskjb4JZbNXCA==", 2024-04-24T23:50:13.453 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.453 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-04-24T23:50:13.453 INFO:teuthology.orchestra.run.smithi016.stdout: "mgr": "allow *", 2024-04-24T23:50:13.453 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow *", 2024-04-24T23:50:13.453 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.454 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-mds", 2024-04-24T23:50:13.454 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQBOmilmrOQyNxAAeZs7pOsehfkmrXfGbyGmJA==", 2024-04-24T23:50:13.454 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.454 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-mds" 2024-04-24T23:50:13.454 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.454 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.454 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.454 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-mgr", 2024-04-24T23:50:13.454 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQBOmilmnO8yNxAADg3QU10V8DqqBeTlM502PA==", 2024-04-24T23:50:13.455 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.455 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-mgr" 2024-04-24T23:50:13.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.455 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-osd", 2024-04-24T23:50:13.455 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQBOmilmYP0yNxAAjy4BO5VNxOyGdW95MZJk5w==", 2024-04-24T23:50:13.455 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.455 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-osd" 2024-04-24T23:50:13.455 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.455 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.455 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.455 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-rbd", 2024-04-24T23:50:13.456 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQBOmilmvwUzNxAAdmndG5WTIC9reznWXqo3XA==", 2024-04-24T23:50:13.456 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.456 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-rbd" 2024-04-24T23:50:13.456 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.456 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-04-24T23:50:13.456 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQBOmilmiQ0zNxAA+EFBPr85nzfxJmq9pHTzQg==", 2024-04-24T23:50:13.456 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.456 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-04-24T23:50:13.456 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.456 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.456 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.457 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "client.bootstrap-rgw", 2024-04-24T23:50:13.457 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQBOmilmRBUzNxAAs7OWL+zTs7iRHeVNs46h5w==", 2024-04-24T23:50:13.457 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.457 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile bootstrap-rgw" 2024-04-24T23:50:13.457 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.457 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.457 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.457 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.x", 2024-04-24T23:50:13.457 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAsmilmKBoWDRAAYsVUTv84qwU3VeIhhLNCTw==", 2024-04-24T23:50:13.457 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.457 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-04-24T23:50:13.457 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-04-24T23:50:13.458 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.458 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:13.458 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:13.458 INFO:teuthology.orchestra.run.smithi016.stdout: "entity": "mgr.y", 2024-04-24T23:50:13.458 INFO:teuthology.orchestra.run.smithi016.stdout: "key": "AQAsmilmrpjUCRAAfH4AjI/pUjQg1/mn7GpDTw==", 2024-04-24T23:50:13.458 INFO:teuthology.orchestra.run.smithi016.stdout: "caps": { 2024-04-24T23:50:13.458 INFO:teuthology.orchestra.run.smithi016.stdout: "mds": "allow *", 2024-04-24T23:50:13.458 INFO:teuthology.orchestra.run.smithi016.stdout: "mon": "allow profile mgr", 2024-04-24T23:50:13.458 INFO:teuthology.orchestra.run.smithi016.stdout: "osd": "allow *" 2024-04-24T23:50:13.458 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.459 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:13.459 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-04-24T23:50:13.459 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:50:13.460 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.442+0000 7f37b67ce640 1 -- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f379c06be30 msgr2=0x7f379c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.442+0000 7f37b67ce640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f379c06be30 0x7f379c06e2f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f37b0072020 tx=0x7f37a4008040 comp rx=0 tx=0).stop 2024-04-24T23:50:13.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.442+0000 7f37b67ce640 1 -- 172.21.15.16:0/704961693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37b00729a0 msgr2=0x7f37b014ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.442+0000 7f37b67ce640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37b00729a0 0x7f37b014ebd0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f37a000c430 tx=0x7f37a000c8f0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.442+0000 7f37b67ce640 1 -- 172.21.15.16:0/704961693 shutdown_connections 2024-04-24T23:50:13.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.442+0000 7f37b67ce640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f379c06be30 0x7f379c06e2f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.442+0000 7f37b67ce640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f37b01621b0 0x7f37b006d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.442+0000 7f37b67ce640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f37b015df40 0x7f37b0157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.442+0000 7f37b67ce640 1 --2- 172.21.15.16:0/704961693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37b00729a0 0x7f37b014ebd0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.442+0000 7f37b67ce640 1 -- 172.21.15.16:0/704961693 >> 172.21.15.16:0/704961693 conn(0x7f37b00d0260 msgr2=0x7f37b0072290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:13.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.442+0000 7f37b67ce640 1 -- 172.21.15.16:0/704961693 shutdown_connections 2024-04-24T23:50:13.461 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.442+0000 7f37b67ce640 1 -- 172.21.15.16:0/704961693 wait complete. 2024-04-24T23:50:13.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-04-24T23:50:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.563+0000 7f46432da640 1 Processor -- start 2024-04-24T23:50:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.564+0000 7f46432da640 1 -- start start 2024-04-24T23:50:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.564+0000 7f46432da640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f463c15df20 0x7f463c071a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.564+0000 7f46432da640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f463c0d0530 0x7f463c071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.564+0000 7f46432da640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f463c0b6a70 con 0x7f463c0cfe50 2024-04-24T23:50:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.565+0000 7f46432da640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f463c0cf460 con 0x7f463c0d0530 2024-04-24T23:50:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.565+0000 7f46432da640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f463c0b6820 con 0x7f463c15df20 2024-04-24T23:50:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.565+0000 7f4640875640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f463c0d0530 0x7f463c071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.565+0000 7f4641877640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f463c0cfe50 0x7f463c0ad8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59030/0 (socket says 172.21.15.16:59030) 2024-04-24T23:50:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.565+0000 7f4641877640 1 -- 172.21.15.16:0/2625963611 learned_addr learned my addr 172.21.15.16:0/2625963611 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.565+0000 7f4641076640 1 --2- 172.21.15.16:0/2625963611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f463c15df20 0x7f463c071a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.565+0000 7f4640875640 1 -- 172.21.15.16:0/2625963611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f463c15df20 msgr2=0x7f463c071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:13.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.565+0000 7f4640875640 1 --2- 172.21.15.16:0/2625963611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f463c15df20 0x7f463c071a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.565+0000 7f4640875640 1 -- 172.21.15.16:0/2625963611 >> v1:172.21.15.16:6789/0 conn(0x7f463c0cfe50 legacy=0x7f463c0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.565+0000 7f4633fff640 1 -- 172.21.15.16:0/2625963611 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2101557993 0 0) 0x7f463c0b6a70 con 0x7f463c0cfe50 2024-04-24T23:50:13.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.565+0000 7f4640875640 1 -- 172.21.15.16:0/2625963611 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c0ce6a0 con 0x7f463c0d0530 2024-04-24T23:50:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.566+0000 7f4640875640 1 --2- 172.21.15.16:0/2625963611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f463c0d0530 0x7f463c071f40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f462c0097a0 tx=0x7f462c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c5cb610a3f7b2a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.566+0000 7f4633fff640 1 -- 172.21.15.16:0/2625963611 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c037070 con 0x7f463c0d0530 2024-04-24T23:50:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.566+0000 7f4633fff640 1 -- 172.21.15.16:0/2625963611 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f462c032440 con 0x7f463c0d0530 2024-04-24T23:50:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.566+0000 7f4633fff640 1 -- 172.21.15.16:0/2625963611 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c032810 con 0x7f463c0d0530 2024-04-24T23:50:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.566+0000 7f46432da640 1 -- 172.21.15.16:0/2625963611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f463c0d0530 msgr2=0x7f463c071f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.566+0000 7f46432da640 1 --2- 172.21.15.16:0/2625963611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f463c0d0530 0x7f463c071f40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f462c0097a0 tx=0x7f462c02a180 comp rx=0 tx=0).stop 2024-04-24T23:50:13.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.566+0000 7f46432da640 1 -- 172.21.15.16:0/2625963611 shutdown_connections 2024-04-24T23:50:13.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.566+0000 7f46432da640 1 --2- 172.21.15.16:0/2625963611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f463c0d0530 0x7f463c071f40 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.566+0000 7f46432da640 1 --2- 172.21.15.16:0/2625963611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f463c15df20 0x7f463c071a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.566+0000 7f46432da640 1 -- 172.21.15.16:0/2625963611 >> 172.21.15.16:0/2625963611 conn(0x7f463c0d0ac0 msgr2=0x7f463c0d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:13.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.567+0000 7f46432da640 1 -- 172.21.15.16:0/2625963611 shutdown_connections 2024-04-24T23:50:13.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.567+0000 7f46432da640 1 -- 172.21.15.16:0/2625963611 wait complete. 2024-04-24T23:50:13.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.568+0000 7f46432da640 1 Processor -- start 2024-04-24T23:50:13.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.568+0000 7f46432da640 1 -- start start 2024-04-24T23:50:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.568+0000 7f46432da640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f463c0cfe50 0x7f463c14c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.568+0000 7f46432da640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f463c0d0530 0x7f463c06dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.568+0000 7f46432da640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f463c15df20 0x7f463c06e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.568+0000 7f46432da640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f463c0cef40 con 0x7f463c15df20 2024-04-24T23:50:13.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.568+0000 7f46432da640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f463c0cec60 con 0x7f463c0d0530 2024-04-24T23:50:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.568+0000 7f46432da640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f463c0b4030 con 0x7f463c0cfe50 2024-04-24T23:50:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f4641877640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f463c15df20 0x7f463c06e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f4641877640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f463c15df20 0x7f463c06e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50724/0 (socket says 172.21.15.16:50724) 2024-04-24T23:50:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f4641877640 1 -- 172.21.15.16:0/2255779869 learned_addr learned my addr 172.21.15.16:0/2255779869 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f4641076640 1 --2- 172.21.15.16:0/2255779869 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f463c0cfe50 0x7f463c14c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f4640875640 1 --2- 172.21.15.16:0/2255779869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f463c0d0530 0x7f463c06dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f4641877640 1 -- 172.21.15.16:0/2255779869 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f463c0cfe50 msgr2=0x7f463c14c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f4641877640 1 --2- 172.21.15.16:0/2255779869 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f463c0cfe50 0x7f463c14c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f4641877640 1 -- 172.21.15.16:0/2255779869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f463c0d0530 msgr2=0x7f463c06dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f4641877640 1 --2- 172.21.15.16:0/2255779869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f463c0d0530 0x7f463c06dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f4641877640 1 -- 172.21.15.16:0/2255779869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463c06c710 con 0x7f463c15df20 2024-04-24T23:50:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f4641877640 1 --2- 172.21.15.16:0/2255779869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f463c15df20 0x7f463c06e420 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f463400c7d0 tx=0x7f463400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f4631ffb640 1 -- 172.21.15.16:0/2255779869 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4634018070 con 0x7f463c15df20 2024-04-24T23:50:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f4631ffb640 1 -- 172.21.15.16:0/2255779869 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f463400a640 con 0x7f463c15df20 2024-04-24T23:50:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f46432da640 1 -- 172.21.15.16:0/2255779869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f463c06c9f0 con 0x7f463c15df20 2024-04-24T23:50:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f4631ffb640 1 -- 172.21.15.16:0/2255779869 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463400aa70 con 0x7f463c15df20 2024-04-24T23:50:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.569+0000 7f46432da640 1 -- 172.21.15.16:0/2255779869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f463c06cfb0 con 0x7f463c15df20 2024-04-24T23:50:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.570+0000 7f46432da640 1 -- 172.21.15.16:0/2255779869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f463c06f8f0 con 0x7f463c15df20 2024-04-24T23:50:13.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.571+0000 7f4631ffb640 1 -- 172.21.15.16:0/2255779869 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f463401b070 con 0x7f463c15df20 2024-04-24T23:50:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.571+0000 7f4631ffb640 1 --2- 172.21.15.16:0/2255779869 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f462006be30 0x7f462006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.571+0000 7f4631ffb640 1 -- 172.21.15.16:0/2255779869 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f463408a260 con 0x7f463c15df20 2024-04-24T23:50:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.571+0000 7f4641076640 1 --2- 172.21.15.16:0/2255779869 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f462006be30 0x7f462006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.572+0000 7f4641076640 1 --2- 172.21.15.16:0/2255779869 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f462006be30 0x7f462006e2f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4638009a30 tx=0x7f4638008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:13.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.576+0000 7f4631ffb640 1 -- 172.21.15.16:0/2255779869 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f4634058e60 con 0x7f463c15df20 2024-04-24T23:50:13.829 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.829+0000 7f46432da640 1 -- 172.21.15.16:0/2255779869 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f463c072b40 con 0x7f463c15df20 2024-04-24T23:50:13.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.831+0000 7f4631ffb640 1 -- 172.21.15.16:0/2255779869 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f463408a080 con 0x7f463c15df20 2024-04-24T23:50:13.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.834+0000 7f46432da640 1 -- 172.21.15.16:0/2255779869 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f462006be30 msgr2=0x7f462006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.834+0000 7f46432da640 1 --2- 172.21.15.16:0/2255779869 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f462006be30 0x7f462006e2f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4638009a30 tx=0x7f4638008040 comp rx=0 tx=0).stop 2024-04-24T23:50:13.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.834+0000 7f46432da640 1 -- 172.21.15.16:0/2255779869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f463c15df20 msgr2=0x7f463c06e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.834+0000 7f46432da640 1 --2- 172.21.15.16:0/2255779869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f463c15df20 0x7f463c06e420 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f463400c7d0 tx=0x7f463400cc90 comp rx=0 tx=0).stop 2024-04-24T23:50:13.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.834+0000 7f46432da640 1 -- 172.21.15.16:0/2255779869 shutdown_connections 2024-04-24T23:50:13.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.834+0000 7f46432da640 1 --2- 172.21.15.16:0/2255779869 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f462006be30 0x7f462006e2f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.834+0000 7f46432da640 1 --2- 172.21.15.16:0/2255779869 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f463c15df20 0x7f463c06e420 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.834+0000 7f46432da640 1 --2- 172.21.15.16:0/2255779869 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f463c0d0530 0x7f463c06dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.834+0000 7f46432da640 1 --2- 172.21.15.16:0/2255779869 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f463c0cfe50 0x7f463c14c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.834+0000 7f46432da640 1 -- 172.21.15.16:0/2255779869 >> 172.21.15.16:0/2255779869 conn(0x7f463c0d0ac0 msgr2=0x7f463c0b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:13.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.834+0000 7f46432da640 1 -- 172.21.15.16:0/2255779869 shutdown_connections 2024-04-24T23:50:13.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.834+0000 7f46432da640 1 -- 172.21.15.16:0/2255779869 wait complete. 2024-04-24T23:50:13.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-04-24T23:50:13.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.946+0000 7f6f88453640 1 Processor -- start 2024-04-24T23:50:13.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.946+0000 7f6f88453640 1 -- start start 2024-04-24T23:50:13.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.947+0000 7f6f88453640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6f80144b20 0x7f6f80144f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.947+0000 7f6f88453640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f80145440 con 0x7f6f800ced90 2024-04-24T23:50:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.947+0000 7f6f88453640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f8015fe90 con 0x7f6f801457b0 2024-04-24T23:50:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.947+0000 7f6f88453640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f6f80160170 con 0x7f6f80144b20 2024-04-24T23:50:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.947+0000 7f6f861ef640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6f80144b20 0x7f6f80144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.947+0000 7f6f869f0640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6f800ced90 0x7f6f8006da40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59038/0 (socket says 172.21.15.16:59038) 2024-04-24T23:50:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.947+0000 7f6f869f0640 1 -- 172.21.15.16:0/715596257 learned_addr learned my addr 172.21.15.16:0/715596257 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.947+0000 7f6f861ef640 1 -- 172.21.15.16:0/715596257 >> v1:172.21.15.156:6789/0 conn(0x7f6f801457b0 legacy=0x7f6f800ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.947+0000 7f6f861ef640 1 -- 172.21.15.16:0/715596257 >> v1:172.21.15.16:6789/0 conn(0x7f6f800ced90 legacy=0x7f6f8006da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.947+0000 7f6f861ef640 1 -- 172.21.15.16:0/715596257 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f80160450 con 0x7f6f80144b20 2024-04-24T23:50:13.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.947+0000 7f6f851ed640 1 -- 172.21.15.16:0/715596257 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3114728639 0 0) 0x7f6f80145440 con 0x7f6f800ced90 2024-04-24T23:50:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.948+0000 7f6f861ef640 1 --2- 172.21.15.16:0/715596257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f80144b20 0x7f6f80144f00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0097a0 tx=0x7f6f7c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb2429dd19e33fae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.948+0000 7f6f851ed640 1 -- 172.21.15.16:0/715596257 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c037070 con 0x7f6f80144b20 2024-04-24T23:50:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.948+0000 7f6f851ed640 1 -- 172.21.15.16:0/715596257 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f7c032440 con 0x7f6f80144b20 2024-04-24T23:50:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.948+0000 7f6f851ed640 1 -- 172.21.15.16:0/715596257 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c02ae60 con 0x7f6f80144b20 2024-04-24T23:50:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.949+0000 7f6f88453640 1 -- 172.21.15.16:0/715596257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f80144b20 msgr2=0x7f6f80144f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.949+0000 7f6f88453640 1 --2- 172.21.15.16:0/715596257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f80144b20 0x7f6f80144f00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0097a0 tx=0x7f6f7c02a110 comp rx=0 tx=0).stop 2024-04-24T23:50:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.949+0000 7f6f88453640 1 -- 172.21.15.16:0/715596257 shutdown_connections 2024-04-24T23:50:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.949+0000 7f6f88453640 1 --2- 172.21.15.16:0/715596257 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f80144b20 0x7f6f80144f00 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.949+0000 7f6f88453640 1 -- 172.21.15.16:0/715596257 >> 172.21.15.16:0/715596257 conn(0x7f6f800b2000 msgr2=0x7f6f800cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.949+0000 7f6f88453640 1 -- 172.21.15.16:0/715596257 shutdown_connections 2024-04-24T23:50:13.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.949+0000 7f6f88453640 1 -- 172.21.15.16:0/715596257 wait complete. 2024-04-24T23:50:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.950+0000 7f6f88453640 1 Processor -- start 2024-04-24T23:50:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.950+0000 7f6f88453640 1 -- start start 2024-04-24T23:50:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f88453640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f800ced90 0x7f6f8013c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f88453640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f80144b20 0x7f6f80141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f88453640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f801457b0 0x7f6f800c6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f88453640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6f800b3390 con 0x7f6f800ced90 2024-04-24T23:50:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f88453640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6f8006d4b0 con 0x7f6f80144b20 2024-04-24T23:50:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f88453640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6f8015f990 con 0x7f6f801457b0 2024-04-24T23:50:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f861ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f800ced90 0x7f6f8013c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f861ef640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f800ced90 0x7f6f8013c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50742/0 (socket says 172.21.15.16:50742) 2024-04-24T23:50:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f861ef640 1 -- 172.21.15.16:0/2609209819 learned_addr learned my addr 172.21.15.16:0/2609209819 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f859ee640 1 --2- 172.21.15.16:0/2609209819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f80144b20 0x7f6f80141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.952 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f869f0640 1 --2- 172.21.15.16:0/2609209819 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f801457b0 0x7f6f800c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f861ef640 1 -- 172.21.15.16:0/2609209819 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f801457b0 msgr2=0x7f6f800c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f861ef640 1 --2- 172.21.15.16:0/2609209819 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f801457b0 0x7f6f800c6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f861ef640 1 -- 172.21.15.16:0/2609209819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f80144b20 msgr2=0x7f6f80141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:13.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f861ef640 1 --2- 172.21.15.16:0/2609209819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f80144b20 0x7f6f80141ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:13.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f861ef640 1 -- 172.21.15.16:0/2609209819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f80140e00 con 0x7f6f800ced90 2024-04-24T23:50:13.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f861ef640 1 --2- 172.21.15.16:0/2609209819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f800ced90 0x7f6f8013c350 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0098c0 tx=0x7f6f7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:13.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f76ffd640 1 -- 172.21.15.16:0/2609209819 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c041070 con 0x7f6f800ced90 2024-04-24T23:50:13.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f76ffd640 1 -- 172.21.15.16:0/2609209819 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f7c03c930 con 0x7f6f800ced90 2024-04-24T23:50:13.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.951+0000 7f6f88453640 1 -- 172.21.15.16:0/2609209819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f801410e0 con 0x7f6f800ced90 2024-04-24T23:50:13.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.952+0000 7f6f76ffd640 1 -- 172.21.15.16:0/2609209819 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c037040 con 0x7f6f800ced90 2024-04-24T23:50:13.953 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.952+0000 7f6f88453640 1 -- 172.21.15.16:0/2609209819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f80141980 con 0x7f6f800ced90 2024-04-24T23:50:13.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.953+0000 7f6f88453640 1 -- 172.21.15.16:0/2609209819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f800731f0 con 0x7f6f800ced90 2024-04-24T23:50:13.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.953+0000 7f6f76ffd640 1 -- 172.21.15.16:0/2609209819 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6f7c044070 con 0x7f6f800ced90 2024-04-24T23:50:13.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.953+0000 7f6f76ffd640 1 --2- 172.21.15.16:0/2609209819 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6f6406be60 0x7f6f6406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:13.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.954+0000 7f6f76ffd640 1 -- 172.21.15.16:0/2609209819 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f6f7c0ab9d0 con 0x7f6f800ced90 2024-04-24T23:50:13.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.954+0000 7f6f859ee640 1 --2- 172.21.15.16:0/2609209819 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6f6406be60 0x7f6f6406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:13.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.954+0000 7f6f859ee640 1 --2- 172.21.15.16:0/2609209819 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6f6406be60 0x7f6f6406e320 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6f800c60a0 tx=0x7f6f70009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:13.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:13.958+0000 7f6f76ffd640 1 -- 172.21.15.16:0/2609209819 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6f7c07a6f0 con 0x7f6f800ced90 2024-04-24T23:50:14.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.210+0000 7f6f88453640 1 -- 172.21.15.16:0/2609209819 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f6f8004f5e0 con 0x7f6f800ced90 2024-04-24T23:50:14.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.212+0000 7f6f76ffd640 1 -- 172.21.15.16:0/2609209819 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7f6f7c079ff0 con 0x7f6f800ced90 2024-04-24T23:50:14.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.215+0000 7f6f88453640 1 -- 172.21.15.16:0/2609209819 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6f6406be60 msgr2=0x7f6f6406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.215+0000 7f6f88453640 1 --2- 172.21.15.16:0/2609209819 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6f6406be60 0x7f6f6406e320 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6f800c60a0 tx=0x7f6f70009040 comp rx=0 tx=0).stop 2024-04-24T23:50:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.215+0000 7f6f88453640 1 -- 172.21.15.16:0/2609209819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f800ced90 msgr2=0x7f6f8013c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.215+0000 7f6f88453640 1 --2- 172.21.15.16:0/2609209819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f800ced90 0x7f6f8013c350 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0098c0 tx=0x7f6f7c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.215+0000 7f6f88453640 1 -- 172.21.15.16:0/2609209819 shutdown_connections 2024-04-24T23:50:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.215+0000 7f6f88453640 1 --2- 172.21.15.16:0/2609209819 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6f6406be60 0x7f6f6406e320 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.215+0000 7f6f88453640 1 --2- 172.21.15.16:0/2609209819 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6f801457b0 0x7f6f800c6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.215+0000 7f6f88453640 1 --2- 172.21.15.16:0/2609209819 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6f80144b20 0x7f6f80141ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.215+0000 7f6f88453640 1 --2- 172.21.15.16:0/2609209819 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6f800ced90 0x7f6f8013c350 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.215+0000 7f6f88453640 1 -- 172.21.15.16:0/2609209819 >> 172.21.15.16:0/2609209819 conn(0x7f6f800b2000 msgr2=0x7f6f8013fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.215+0000 7f6f88453640 1 -- 172.21.15.16:0/2609209819 shutdown_connections 2024-04-24T23:50:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.215+0000 7f6f88453640 1 -- 172.21.15.16:0/2609209819 wait complete. 2024-04-24T23:50:14.226 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-04-24T23:50:14.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.328+0000 7f70302a5640 1 Processor -- start 2024-04-24T23:50:14.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.328+0000 7f70302a5640 1 -- start start 2024-04-24T23:50:14.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f70302a5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f702815dfa0 0x7f70280d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f70302a5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70280ce5f0 0x7f70280cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f70302a5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f70280b5f00 0x7f70280b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f70302a5640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f70280b7360 con 0x7f70280ce5f0 2024-04-24T23:50:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f70302a5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f70280cc640 con 0x7f70280b5f00 2024-04-24T23:50:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f70302a5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f70280d05a0 con 0x7f702815dfa0 2024-04-24T23:50:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f702d840640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70280ce5f0 0x7f70280cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f702d840640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f70280ce5f0 0x7f70280cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50758/0 (socket says 172.21.15.16:50758) 2024-04-24T23:50:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f702d840640 1 -- 172.21.15.16:0/1438595456 learned_addr learned my addr 172.21.15.16:0/1438595456 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f702e041640 1 --2- 172.21.15.16:0/1438595456 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f702815dfa0 0x7f70280d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f702e842640 1 --2- 172.21.15.16:0/1438595456 >> v2:172.21.15.156:3300/0 conn(0x7f70280b5f00 0x7f70280b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f702d840640 1 -- 172.21.15.16:0/1438595456 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f702815dfa0 msgr2=0x7f70280d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:14.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f702d840640 1 --2- 172.21.15.16:0/1438595456 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f702815dfa0 0x7f70280d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f702d840640 1 -- 172.21.15.16:0/1438595456 >> v2:172.21.15.156:3300/0 conn(0x7f70280b5f00 msgr2=0x7f70280b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f702d840640 1 --2- 172.21.15.16:0/1438595456 >> v2:172.21.15.156:3300/0 conn(0x7f70280b5f00 0x7f70280b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f702d840640 1 -- 172.21.15.16:0/1438595456 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70280d0880 con 0x7f70280ce5f0 2024-04-24T23:50:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.329+0000 7f702e842640 1 --2- 172.21.15.16:0/1438595456 >> v2:172.21.15.156:3300/0 conn(0x7f70280b5f00 0x7f70280b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.330+0000 7f702d840640 1 --2- 172.21.15.16:0/1438595456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70280ce5f0 0x7f70280cea70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f70280cc1e0 tx=0x7f701802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3882d46348810c87 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.330+0000 7f702d03f640 1 -- 172.21.15.16:0/1438595456 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7018037070 con 0x7f70280ce5f0 2024-04-24T23:50:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.330+0000 7f702d03f640 1 -- 172.21.15.16:0/1438595456 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7018032440 con 0x7f70280ce5f0 2024-04-24T23:50:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.330+0000 7f702d03f640 1 -- 172.21.15.16:0/1438595456 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7018032810 con 0x7f70280ce5f0 2024-04-24T23:50:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.331+0000 7f70302a5640 1 -- 172.21.15.16:0/1438595456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70280ce5f0 msgr2=0x7f70280cea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.331+0000 7f70302a5640 1 --2- 172.21.15.16:0/1438595456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70280ce5f0 0x7f70280cea70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f70280cc1e0 tx=0x7f701802a1e0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.331+0000 7f70302a5640 1 -- 172.21.15.16:0/1438595456 shutdown_connections 2024-04-24T23:50:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.331+0000 7f70302a5640 1 --2- 172.21.15.16:0/1438595456 >> v2:172.21.15.156:3300/0 conn(0x7f70280b5f00 0x7f70280b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.331+0000 7f70302a5640 1 --2- 172.21.15.16:0/1438595456 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f70280ce5f0 0x7f70280cea70 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.331+0000 7f70302a5640 1 --2- 172.21.15.16:0/1438595456 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f702815dfa0 0x7f70280d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.331+0000 7f70302a5640 1 -- 172.21.15.16:0/1438595456 >> 172.21.15.16:0/1438595456 conn(0x7f702806b4d0 msgr2=0x7f70280b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.331+0000 7f70302a5640 1 -- 172.21.15.16:0/1438595456 shutdown_connections 2024-04-24T23:50:14.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.331+0000 7f70302a5640 1 -- 172.21.15.16:0/1438595456 wait complete. 2024-04-24T23:50:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.332+0000 7f70302a5640 1 Processor -- start 2024-04-24T23:50:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.332+0000 7f70302a5640 1 -- start start 2024-04-24T23:50:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f70302a5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70280b5f00 0x7f7028070380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f70302a5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70280ce5f0 0x7f70280c7910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f70302a5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f702815dfa0 0x7f702806d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:14.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f70302a5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f702815e380 con 0x7f702815dfa0 2024-04-24T23:50:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f70302a5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f70280cc2d0 con 0x7f70280b5f00 2024-04-24T23:50:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f70302a5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f70280d1520 con 0x7f70280ce5f0 2024-04-24T23:50:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f702e842640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f702815dfa0 0x7f702806d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f702e842640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f702815dfa0 0x7f702806d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50766/0 (socket says 172.21.15.16:50766) 2024-04-24T23:50:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f702e842640 1 -- 172.21.15.16:0/1133828644 learned_addr learned my addr 172.21.15.16:0/1133828644 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f702d840640 1 --2- 172.21.15.16:0/1133828644 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70280ce5f0 0x7f70280c7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f702e041640 1 --2- 172.21.15.16:0/1133828644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70280b5f00 0x7f7028070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f702e842640 1 -- 172.21.15.16:0/1133828644 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70280ce5f0 msgr2=0x7f70280c7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f702e842640 1 --2- 172.21.15.16:0/1133828644 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70280ce5f0 0x7f70280c7910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f702e842640 1 -- 172.21.15.16:0/1133828644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70280b5f00 msgr2=0x7f7028070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:14.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f702e842640 1 --2- 172.21.15.16:0/1133828644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70280b5f00 0x7f7028070380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f702e842640 1 -- 172.21.15.16:0/1133828644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70280708c0 con 0x7f702815dfa0 2024-04-24T23:50:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f702e842640 1 --2- 172.21.15.16:0/1133828644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f702815dfa0 0x7f702806d490 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7020007e10 tx=0x7f702000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f701effd640 1 -- 172.21.15.16:0/1133828644 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f702001a070 con 0x7f702815dfa0 2024-04-24T23:50:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.333+0000 7f701effd640 1 -- 172.21.15.16:0/1133828644 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7020003b90 con 0x7f702815dfa0 2024-04-24T23:50:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.334+0000 7f701effd640 1 -- 172.21.15.16:0/1133828644 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7020004030 con 0x7f702815dfa0 2024-04-24T23:50:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.334+0000 7f70302a5640 1 -- 172.21.15.16:0/1133828644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70280c6870 con 0x7f702815dfa0 2024-04-24T23:50:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.334+0000 7f70302a5640 1 -- 172.21.15.16:0/1133828644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70280c7110 con 0x7f702815dfa0 2024-04-24T23:50:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.335+0000 7f70302a5640 1 -- 172.21.15.16:0/1133828644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f702806bea0 con 0x7f702815dfa0 2024-04-24T23:50:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.335+0000 7f701effd640 1 -- 172.21.15.16:0/1133828644 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f702001d070 con 0x7f702815dfa0 2024-04-24T23:50:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.335+0000 7f701effd640 1 --2- 172.21.15.16:0/1133828644 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f700c06be30 0x7f700c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.336+0000 7f701effd640 1 -- 172.21.15.16:0/1133828644 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f702008af60 con 0x7f702815dfa0 2024-04-24T23:50:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.336+0000 7f702e041640 1 --2- 172.21.15.16:0/1133828644 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f700c06be30 0x7f700c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:14.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.336+0000 7f702e041640 1 --2- 172.21.15.16:0/1133828644 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f700c06be30 0x7f700c06e2f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7024009970 tx=0x7f7024008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:14.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.340+0000 7f701effd640 1 -- 172.21.15.16:0/1133828644 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7020010070 con 0x7f702815dfa0 2024-04-24T23:50:14.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.594+0000 7f70302a5640 1 -- 172.21.15.16:0/1133828644 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f70280c4e40 con 0x7f702815dfa0 2024-04-24T23:50:14.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.596+0000 7f701effd640 1 -- 172.21.15.16:0/1133828644 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f7020059ad0 con 0x7f702815dfa0 2024-04-24T23:50:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.598+0000 7f70302a5640 1 -- 172.21.15.16:0/1133828644 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f700c06be30 msgr2=0x7f700c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.598+0000 7f70302a5640 1 --2- 172.21.15.16:0/1133828644 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f700c06be30 0x7f700c06e2f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7024009970 tx=0x7f7024008040 comp rx=0 tx=0).stop 2024-04-24T23:50:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.598+0000 7f70302a5640 1 -- 172.21.15.16:0/1133828644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f702815dfa0 msgr2=0x7f702806d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.598+0000 7f70302a5640 1 --2- 172.21.15.16:0/1133828644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f702815dfa0 0x7f702806d490 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f7020007e10 tx=0x7f702000d4a0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.598+0000 7f70302a5640 1 -- 172.21.15.16:0/1133828644 shutdown_connections 2024-04-24T23:50:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.598+0000 7f70302a5640 1 --2- 172.21.15.16:0/1133828644 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f700c06be30 0x7f700c06e2f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.598+0000 7f70302a5640 1 --2- 172.21.15.16:0/1133828644 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f702815dfa0 0x7f702806d490 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.598+0000 7f70302a5640 1 --2- 172.21.15.16:0/1133828644 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f70280ce5f0 0x7f70280c7910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.598+0000 7f70302a5640 1 --2- 172.21.15.16:0/1133828644 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f70280b5f00 0x7f7028070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.598+0000 7f70302a5640 1 -- 172.21.15.16:0/1133828644 >> 172.21.15.16:0/1133828644 conn(0x7f702806b4d0 msgr2=0x7f70281607f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.598+0000 7f70302a5640 1 -- 172.21.15.16:0/1133828644 shutdown_connections 2024-04-24T23:50:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.598+0000 7f70302a5640 1 -- 172.21.15.16:0/1133828644 wait complete. 2024-04-24T23:50:14.609 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-04-24T23:50:14.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.711+0000 7fb171c93640 1 Processor -- start 2024-04-24T23:50:14.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.711+0000 7fb171c93640 1 -- start start 2024-04-24T23:50:14.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.712+0000 7fb171c93640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb16c0ce5f0 0x7fb16c0cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:14.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.712+0000 7fb171c93640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb16c0ccad0 0x7fb16c0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:14.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.712+0000 7fb171c93640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb16c0d05a0 con 0x7fb16c15dfa0 2024-04-24T23:50:14.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.712+0000 7fb171c93640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb16c0b7360 con 0x7fb16c0ccad0 2024-04-24T23:50:14.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.712+0000 7fb171c93640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb16c0cc640 con 0x7fb16c0ce5f0 2024-04-24T23:50:14.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.712+0000 7fb16bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb16c0ccad0 0x7fb16c0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:14.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.712+0000 7fb16bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb16c0ccad0 0x7fb16c0ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50778/0 (socket says 172.21.15.16:50778) 2024-04-24T23:50:14.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.712+0000 7fb16bfff640 1 -- 172.21.15.16:0/1971108645 learned_addr learned my addr 172.21.15.16:0/1971108645 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:14.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.712+0000 7fb16affd640 1 --2- 172.21.15.16:0/1971108645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb16c0ce5f0 0x7fb16c0cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:14.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.712+0000 7fb16bfff640 1 -- 172.21.15.16:0/1971108645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb16c0ce5f0 msgr2=0x7fb16c0cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:14.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.712+0000 7fb16bfff640 1 --2- 172.21.15.16:0/1971108645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb16c0ce5f0 0x7fb16c0cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.712+0000 7fb16bfff640 1 -- 172.21.15.16:0/1971108645 >> v1:172.21.15.156:6789/0 conn(0x7fb16c15dfa0 legacy=0x7fb16c0d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:14.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.712+0000 7fb16bfff640 1 -- 172.21.15.16:0/1971108645 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16c0d0880 con 0x7fb16c0ccad0 2024-04-24T23:50:14.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.713+0000 7fb16bfff640 1 --2- 172.21.15.16:0/1971108645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb16c0ccad0 0x7fb16c0ccf60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb164009e50 tx=0x7fb16402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70594198a65fbb11 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.713+0000 7fb16a7fc640 1 -- 172.21.15.16:0/1971108645 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb164036070 con 0x7fb16c0ccad0 2024-04-24T23:50:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.713+0000 7fb16a7fc640 1 -- 172.21.15.16:0/1971108645 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb164031600 con 0x7fb16c0ccad0 2024-04-24T23:50:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.713+0000 7fb16a7fc640 1 -- 172.21.15.16:0/1971108645 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1640319d0 con 0x7fb16c0ccad0 2024-04-24T23:50:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.714+0000 7fb171c93640 1 -- 172.21.15.16:0/1971108645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb16c0ccad0 msgr2=0x7fb16c0ccf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.714+0000 7fb171c93640 1 --2- 172.21.15.16:0/1971108645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb16c0ccad0 0x7fb16c0ccf60 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb164009e50 tx=0x7fb16402a710 comp rx=0 tx=0).stop 2024-04-24T23:50:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.714+0000 7fb171c93640 1 -- 172.21.15.16:0/1971108645 shutdown_connections 2024-04-24T23:50:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.714+0000 7fb171c93640 1 --2- 172.21.15.16:0/1971108645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb16c0ce5f0 0x7fb16c0cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.714+0000 7fb171c93640 1 --2- 172.21.15.16:0/1971108645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb16c0ccad0 0x7fb16c0ccf60 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.714+0000 7fb171c93640 1 -- 172.21.15.16:0/1971108645 >> 172.21.15.16:0/1971108645 conn(0x7fb16c06b4d0 msgr2=0x7fb16c0b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.714+0000 7fb171c93640 1 -- 172.21.15.16:0/1971108645 shutdown_connections 2024-04-24T23:50:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.714+0000 7fb171c93640 1 -- 172.21.15.16:0/1971108645 wait complete. 2024-04-24T23:50:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.715+0000 7fb171c93640 1 Processor -- start 2024-04-24T23:50:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.715+0000 7fb171c93640 1 -- start start 2024-04-24T23:50:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb171c93640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb16c0ccad0 0x7fb16c151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb171c93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb16c0ce5f0 0x7fb16c0c5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb171c93640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb16c15dfa0 0x7fb16c15c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb171c93640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c15e380 con 0x7fb16c0ce5f0 2024-04-24T23:50:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb171c93640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb16c0cc2d0 con 0x7fb16c0ccad0 2024-04-24T23:50:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb171c93640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb16c0d01c0 con 0x7fb16c15dfa0 2024-04-24T23:50:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb16affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb16c0ce5f0 0x7fb16c0c5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb16affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb16c0ce5f0 0x7fb16c0c5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50782/0 (socket says 172.21.15.16:50782) 2024-04-24T23:50:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb16affd640 1 -- 172.21.15.16:0/1068165057 learned_addr learned my addr 172.21.15.16:0/1068165057 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb16bfff640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb16c15dfa0 0x7fb16c15c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb16b7fe640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb16c0ccad0 0x7fb16c151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb16affd640 1 -- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb16c15dfa0 msgr2=0x7fb16c15c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb16affd640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb16c15dfa0 0x7fb16c15c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb16affd640 1 -- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb16c0ccad0 msgr2=0x7fb16c151000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.716+0000 7fb16affd640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb16c0ccad0 0x7fb16c151000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.717+0000 7fb16affd640 1 -- 172.21.15.16:0/1068165057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16c0c6780 con 0x7fb16c0ce5f0 2024-04-24T23:50:14.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.717+0000 7fb16bfff640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb16c15dfa0 0x7fb16c15c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:14.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.717+0000 7fb16affd640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb16c0ce5f0 0x7fb16c0c5c80 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb16000e9e0 tx=0x7fb16000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:14.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.717+0000 7fb153fff640 1 -- 172.21.15.16:0/1068165057 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160017070 con 0x7fb16c0ce5f0 2024-04-24T23:50:14.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.717+0000 7fb153fff640 1 -- 172.21.15.16:0/1068165057 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb16000c970 con 0x7fb16c0ce5f0 2024-04-24T23:50:14.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.717+0000 7fb171c93640 1 -- 172.21.15.16:0/1068165057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb16c0c6a60 con 0x7fb16c0ce5f0 2024-04-24T23:50:14.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.717+0000 7fb153fff640 1 -- 172.21.15.16:0/1068165057 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb16000cd60 con 0x7fb16c0ce5f0 2024-04-24T23:50:14.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.717+0000 7fb171c93640 1 -- 172.21.15.16:0/1068165057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb16c0c7020 con 0x7fb16c0ce5f0 2024-04-24T23:50:14.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.718+0000 7fb171c93640 1 -- 172.21.15.16:0/1068165057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb16c06bea0 con 0x7fb16c0ce5f0 2024-04-24T23:50:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.718+0000 7fb153fff640 1 -- 172.21.15.16:0/1068165057 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb16001a070 con 0x7fb16c0ce5f0 2024-04-24T23:50:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.719+0000 7fb153fff640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb15806be30 0x7fb15806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.719+0000 7fb153fff640 1 -- 172.21.15.16:0/1068165057 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fb16008af90 con 0x7fb16c0ce5f0 2024-04-24T23:50:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.719+0000 7fb16b7fe640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb15806be30 0x7fb15806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:14.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.720+0000 7fb16b7fe640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb15806be30 0x7fb15806e2f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb15c001500 tx=0x7fb15c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:14.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.723+0000 7fb153fff640 1 -- 172.21.15.16:0/1068165057 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb160059b90 con 0x7fb16c0ce5f0 2024-04-24T23:50:14.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.980+0000 7fb171c93640 1 -- 172.21.15.16:0/1068165057 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7fb16c04f5e0 con 0x7fb16c0ce5f0 2024-04-24T23:50:14.981 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.982+0000 7fb153fff640 1 -- 172.21.15.16:0/1068165057 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7fb160059490 con 0x7fb16c0ce5f0 2024-04-24T23:50:14.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.984+0000 7fb171c93640 1 -- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb15806be30 msgr2=0x7fb15806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:14.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.984+0000 7fb171c93640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb15806be30 0x7fb15806e2f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb15c001500 tx=0x7fb15c009040 comp rx=0 tx=0).stop 2024-04-24T23:50:14.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.984+0000 7fb171c93640 1 -- 172.21.15.16:0/1068165057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb16c0ce5f0 msgr2=0x7fb16c0c5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:14.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.984+0000 7fb171c93640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb16c0ce5f0 0x7fb16c0c5c80 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb16000e9e0 tx=0x7fb16000eea0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.984+0000 7fb171c93640 1 -- 172.21.15.16:0/1068165057 shutdown_connections 2024-04-24T23:50:14.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.984+0000 7fb171c93640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb15806be30 0x7fb15806e2f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.984+0000 7fb171c93640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb16c15dfa0 0x7fb16c15c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.984+0000 7fb171c93640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb16c0ce5f0 0x7fb16c0c5c80 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.984+0000 7fb171c93640 1 --2- 172.21.15.16:0/1068165057 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb16c0ccad0 0x7fb16c151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:14.984 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.984+0000 7fb171c93640 1 -- 172.21.15.16:0/1068165057 >> 172.21.15.16:0/1068165057 conn(0x7fb16c06b4d0 msgr2=0x7fb16c0b5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:14.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.984+0000 7fb171c93640 1 -- 172.21.15.16:0/1068165057 shutdown_connections 2024-04-24T23:50:14.985 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:14.984+0000 7fb171c93640 1 -- 172.21.15.16:0/1068165057 wait complete. 2024-04-24T23:50:14.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-04-24T23:50:15.096 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.096+0000 7f0317342640 1 Processor -- start 2024-04-24T23:50:15.126 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.096+0000 7f0317342640 1 -- start start 2024-04-24T23:50:15.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.097+0000 7f0317342640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0310144b20 0x7f0310144f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.097+0000 7f0317342640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03100ced90 0x7f031006da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.097+0000 7f0317342640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0310145440 con 0x7f03101457b0 2024-04-24T23:50:15.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.097+0000 7f0317342640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f031015fe90 con 0x7f03100ced90 2024-04-24T23:50:15.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.097+0000 7f0317342640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0310160170 con 0x7f0310144b20 2024-04-24T23:50:15.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.097+0000 7f03158df640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03100ced90 0x7f031006da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.097+0000 7f03148dd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f03101457b0 0x7f03100ce680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59052/0 (socket says 172.21.15.16:59052) 2024-04-24T23:50:15.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.097+0000 7f03158df640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03100ced90 0x7f031006da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50792/0 (socket says 172.21.15.16:50792) 2024-04-24T23:50:15.131 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.097+0000 7f03158df640 1 -- 172.21.15.16:0/1519380601 learned_addr learned my addr 172.21.15.16:0/1519380601 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.098+0000 7f03150de640 1 --2- 172.21.15.16:0/1519380601 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0310144b20 0x7f0310144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.098+0000 7f03158df640 1 -- 172.21.15.16:0/1519380601 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0310144b20 msgr2=0x7f0310144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.098+0000 7f03158df640 1 --2- 172.21.15.16:0/1519380601 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0310144b20 0x7f0310144f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.098+0000 7f03158df640 1 -- 172.21.15.16:0/1519380601 >> v1:172.21.15.16:6789/0 conn(0x7f03101457b0 legacy=0x7f03100ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.098+0000 7f03158df640 1 -- 172.21.15.16:0/1519380601 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0310160450 con 0x7f03100ced90 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.098+0000 7f0307fff640 1 -- 172.21.15.16:0/1519380601 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1714635814 0 0) 0x7f0310145440 con 0x7f03101457b0 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.098+0000 7f03158df640 1 --2- 172.21.15.16:0/1519380601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100ced90 0x7f031006da40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f030c009e50 tx=0x7f030c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92f225b4634ca60 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.099+0000 7f0307fff640 1 -- 172.21.15.16:0/1519380601 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030c036070 con 0x7f03100ced90 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.099+0000 7f0307fff640 1 -- 172.21.15.16:0/1519380601 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f030c031600 con 0x7f03100ced90 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.099+0000 7f0307fff640 1 -- 172.21.15.16:0/1519380601 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030c0319d0 con 0x7f03100ced90 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.099+0000 7f0317342640 1 -- 172.21.15.16:0/1519380601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100ced90 msgr2=0x7f031006da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.099+0000 7f0317342640 1 --2- 172.21.15.16:0/1519380601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100ced90 0x7f031006da40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f030c009e50 tx=0x7f030c02a680 comp rx=0 tx=0).stop 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.099+0000 7f0317342640 1 -- 172.21.15.16:0/1519380601 shutdown_connections 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.099+0000 7f0317342640 1 --2- 172.21.15.16:0/1519380601 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100ced90 0x7f031006da40 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.132 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.099+0000 7f0317342640 1 --2- 172.21.15.16:0/1519380601 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0310144b20 0x7f0310144f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.099+0000 7f0317342640 1 -- 172.21.15.16:0/1519380601 >> 172.21.15.16:0/1519380601 conn(0x7f03100b2000 msgr2=0x7f03100cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:15.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.099+0000 7f0317342640 1 -- 172.21.15.16:0/1519380601 shutdown_connections 2024-04-24T23:50:15.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.099+0000 7f0317342640 1 -- 172.21.15.16:0/1519380601 wait complete. 2024-04-24T23:50:15.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.100+0000 7f0317342640 1 Processor -- start 2024-04-24T23:50:15.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.100+0000 7f0317342640 1 -- start start 2024-04-24T23:50:15.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.101+0000 7f0317342640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100ced90 0x7f031013dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.101+0000 7f0317342640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0310144b20 0x7f031015eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.101+0000 7f0317342640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03101457b0 0x7f0310143370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.101+0000 7f0317342640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f03100b3390 con 0x7f03100ced90 2024-04-24T23:50:15.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.101+0000 7f0317342640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f031006d4b0 con 0x7f0310144b20 2024-04-24T23:50:15.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.101+0000 7f0317342640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f031015f990 con 0x7f03101457b0 2024-04-24T23:50:15.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.101+0000 7f03150de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100ced90 0x7f031013dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.101+0000 7f03150de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100ced90 0x7f031013dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50800/0 (socket says 172.21.15.16:50800) 2024-04-24T23:50:15.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.101+0000 7f03150de640 1 -- 172.21.15.16:0/2242569040 learned_addr learned my addr 172.21.15.16:0/2242569040 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:15.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f03158df640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03101457b0 0x7f0310143370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f03148dd640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0310144b20 0x7f031015eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f03150de640 1 -- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03101457b0 msgr2=0x7f0310143370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:15.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f03150de640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03101457b0 0x7f0310143370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f03150de640 1 -- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0310144b20 msgr2=0x7f031015eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:15.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f03150de640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0310144b20 0x7f031015eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f03150de640 1 -- 172.21.15.16:0/2242569040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f031013e260 con 0x7f03100ced90 2024-04-24T23:50:15.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f03158df640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03101457b0 0x7f0310143370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:15.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f03148dd640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0310144b20 0x7f031015eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:15.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f03150de640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100ced90 0x7f031013dd20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f030800d830 tx=0x7f030800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:15.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f0305ffb640 1 -- 172.21.15.16:0/2242569040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030800b6c0 con 0x7f03100ced90 2024-04-24T23:50:15.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f0305ffb640 1 -- 172.21.15.16:0/2242569040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0308019070 con 0x7f03100ced90 2024-04-24T23:50:15.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f0317342640 1 -- 172.21.15.16:0/2242569040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03100cd450 con 0x7f03100ced90 2024-04-24T23:50:15.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f0305ffb640 1 -- 172.21.15.16:0/2242569040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f030800ba80 con 0x7f03100ced90 2024-04-24T23:50:15.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.102+0000 7f0317342640 1 -- 172.21.15.16:0/2242569040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03101f77a0 con 0x7f03100ced90 2024-04-24T23:50:15.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.103+0000 7f0317342640 1 -- 172.21.15.16:0/2242569040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03100716a0 con 0x7f03100ced90 2024-04-24T23:50:15.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.104+0000 7f0305ffb640 1 -- 172.21.15.16:0/2242569040 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f030801e460 con 0x7f03100ced90 2024-04-24T23:50:15.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.104+0000 7f0305ffb640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02f406be30 0x7f02f406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.104+0000 7f0305ffb640 1 -- 172.21.15.16:0/2242569040 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f030808b2f0 con 0x7f03100ced90 2024-04-24T23:50:15.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.104+0000 7f03148dd640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02f406be30 0x7f02f406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.105+0000 7f03148dd640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02f406be30 0x7f02f406e2f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f031015ea90 tx=0x7f0300009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:15.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.109+0000 7f0305ffb640 1 -- 172.21.15.16:0/2242569040 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f03100716a0 con 0x7f03100ced90 2024-04-24T23:50:15.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.362+0000 7f0317342640 1 -- 172.21.15.16:0/2242569040 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7f03100731f0 con 0x7f03100ced90 2024-04-24T23:50:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.364+0000 7f0305ffb640 1 -- 172.21.15.16:0/2242569040 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7f0308059ef0 con 0x7f03100ced90 2024-04-24T23:50:15.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.367+0000 7f0317342640 1 -- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02f406be30 msgr2=0x7f02f406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:15.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.367+0000 7f0317342640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02f406be30 0x7f02f406e2f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f031015ea90 tx=0x7f0300009040 comp rx=0 tx=0).stop 2024-04-24T23:50:15.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.367+0000 7f0317342640 1 -- 172.21.15.16:0/2242569040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100ced90 msgr2=0x7f031013dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:15.367 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.367+0000 7f0317342640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100ced90 0x7f031013dd20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f030800d830 tx=0x7f030800dcf0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.367+0000 7f0317342640 1 -- 172.21.15.16:0/2242569040 shutdown_connections 2024-04-24T23:50:15.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.367+0000 7f0317342640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f02f406be30 0x7f02f406e2f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.367+0000 7f0317342640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f03101457b0 0x7f0310143370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.367+0000 7f0317342640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0310144b20 0x7f031015eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.367+0000 7f0317342640 1 --2- 172.21.15.16:0/2242569040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f03100ced90 0x7f031013dd20 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.367+0000 7f0317342640 1 -- 172.21.15.16:0/2242569040 >> 172.21.15.16:0/2242569040 conn(0x7f03100b2000 msgr2=0x7f03100718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:15.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.367+0000 7f0317342640 1 -- 172.21.15.16:0/2242569040 shutdown_connections 2024-04-24T23:50:15.368 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.367+0000 7f0317342640 1 -- 172.21.15.16:0/2242569040 wait complete. 2024-04-24T23:50:15.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-04-24T23:50:15.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.481+0000 7f72612cd640 1 Processor -- start 2024-04-24T23:50:15.481 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.481+0000 7f72612cd640 1 -- start start 2024-04-24T23:50:15.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.482+0000 7f72612cd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f725c0af940 0x7f725c0adc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.482+0000 7f72612cd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f725c1622d0 0x7f725c0b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.482+0000 7f72612cd640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f725c147b90 con 0x7f725c1472f0 2024-04-24T23:50:15.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.482+0000 7f72612cd640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f725c162b70 con 0x7f725c0af940 2024-04-24T23:50:15.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.482+0000 7f72612cd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f725c0b0100 con 0x7f725c1622d0 2024-04-24T23:50:15.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.482+0000 7f725affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f725c0af940 0x7f725c0adc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.482+0000 7f725affd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f725c0af940 0x7f725c0adc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50814/0 (socket says 172.21.15.16:50814) 2024-04-24T23:50:15.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.482+0000 7f725affd640 1 -- 172.21.15.16:0/687419636 learned_addr learned my addr 172.21.15.16:0/687419636 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:15.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.482+0000 7f725b7fe640 1 --2- 172.21.15.16:0/687419636 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f725c1622d0 0x7f725c0b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.482+0000 7f725affd640 1 -- 172.21.15.16:0/687419636 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f725c1622d0 msgr2=0x7f725c0b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:15.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.482+0000 7f725affd640 1 --2- 172.21.15.16:0/687419636 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f725c1622d0 0x7f725c0b64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.482+0000 7f725affd640 1 -- 172.21.15.16:0/687419636 >> v1:172.21.15.156:6789/0 conn(0x7f725c1472f0 legacy=0x7f725c161bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:15.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.482+0000 7f725affd640 1 -- 172.21.15.16:0/687419636 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725c0afd20 con 0x7f725c0af940 2024-04-24T23:50:15.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.482+0000 7f725b7fe640 1 --2- 172.21.15.16:0/687419636 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f725c1622d0 0x7f725c0b64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:15.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.483+0000 7f725affd640 1 --2- 172.21.15.16:0/687419636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f725c0af940 0x7f725c0adc80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f724c009870 tx=0x7f724c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4df8b204ccfdde6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:15.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.483+0000 7f725a7fc640 1 -- 172.21.15.16:0/687419636 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c037070 con 0x7f725c0af940 2024-04-24T23:50:15.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.483+0000 7f725a7fc640 1 -- 172.21.15.16:0/687419636 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f724c032440 con 0x7f725c0af940 2024-04-24T23:50:15.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.483+0000 7f725a7fc640 1 -- 172.21.15.16:0/687419636 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c032810 con 0x7f725c0af940 2024-04-24T23:50:15.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.483+0000 7f72612cd640 1 -- 172.21.15.16:0/687419636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f725c0af940 msgr2=0x7f725c0adc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:15.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.483+0000 7f72612cd640 1 --2- 172.21.15.16:0/687419636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f725c0af940 0x7f725c0adc80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f724c009870 tx=0x7f724c02a180 comp rx=0 tx=0).stop 2024-04-24T23:50:15.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.484+0000 7f72612cd640 1 -- 172.21.15.16:0/687419636 shutdown_connections 2024-04-24T23:50:15.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.484+0000 7f72612cd640 1 --2- 172.21.15.16:0/687419636 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f725c1622d0 0x7f725c0b64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.484+0000 7f72612cd640 1 --2- 172.21.15.16:0/687419636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f725c0af940 0x7f725c0adc80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.484+0000 7f72612cd640 1 -- 172.21.15.16:0/687419636 >> 172.21.15.16:0/687419636 conn(0x7f725c071b40 msgr2=0x7f725c071f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:15.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.484+0000 7f72612cd640 1 -- 172.21.15.16:0/687419636 shutdown_connections 2024-04-24T23:50:15.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.484+0000 7f72612cd640 1 -- 172.21.15.16:0/687419636 wait complete. 2024-04-24T23:50:15.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.485+0000 7f72612cd640 1 Processor -- start 2024-04-24T23:50:15.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.485+0000 7f72612cd640 1 -- start start 2024-04-24T23:50:15.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f72612cd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f725c0af940 0x7f725c13e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f72612cd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f725c1472f0 0x7f725c15eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f72612cd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f725c1622d0 0x7f725c15eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f72612cd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f725c0b0680 con 0x7f725c0af940 2024-04-24T23:50:15.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f72612cd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f725c0b03a0 con 0x7f725c1472f0 2024-04-24T23:50:15.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f72612cd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f725c0cf7f0 con 0x7f725c1622d0 2024-04-24T23:50:15.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f725affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f725c0af940 0x7f725c13e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f725affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f725c0af940 0x7f725c13e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50822/0 (socket says 172.21.15.16:50822) 2024-04-24T23:50:15.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f725affd640 1 -- 172.21.15.16:0/20058647 learned_addr learned my addr 172.21.15.16:0/20058647 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:15.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f72527fc640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f725c1472f0 0x7f725c15eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f725b7fe640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f725c1622d0 0x7f725c15eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f725affd640 1 -- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f725c1622d0 msgr2=0x7f725c15eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:15.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f725affd640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f725c1622d0 0x7f725c15eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f725affd640 1 -- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f725c1472f0 msgr2=0x7f725c15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:15.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f725affd640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f725c1472f0 0x7f725c15eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f725affd640 1 -- 172.21.15.16:0/20058647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f725c06c490 con 0x7f725c0af940 2024-04-24T23:50:15.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f72527fc640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f725c1472f0 0x7f725c15eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:15.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f725b7fe640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f725c1622d0 0x7f725c15eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:15.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f725affd640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f725c0af940 0x7f725c13e740 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f724c009870 tx=0x7f724c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:15.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f7253fff640 1 -- 172.21.15.16:0/20058647 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c041020 con 0x7f725c0af940 2024-04-24T23:50:15.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f7253fff640 1 -- 172.21.15.16:0/20058647 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f724c030970 con 0x7f725c0af940 2024-04-24T23:50:15.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f72612cd640 1 -- 172.21.15.16:0/20058647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f725c06c770 con 0x7f725c0af940 2024-04-24T23:50:15.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f7253fff640 1 -- 172.21.15.16:0/20058647 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c037040 con 0x7f725c0af940 2024-04-24T23:50:15.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.486+0000 7f72612cd640 1 -- 172.21.15.16:0/20058647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f725c06cd30 con 0x7f725c0af940 2024-04-24T23:50:15.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.487+0000 7f72612cd640 1 -- 172.21.15.16:0/20058647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f725c074a20 con 0x7f725c0af940 2024-04-24T23:50:15.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.488+0000 7f7253fff640 1 -- 172.21.15.16:0/20058647 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f724c044020 con 0x7f725c0af940 2024-04-24T23:50:15.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.488+0000 7f7253fff640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f723c06be30 0x7f723c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.488+0000 7f7253fff640 1 -- 172.21.15.16:0/20058647 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f724c0ab670 con 0x7f725c0af940 2024-04-24T23:50:15.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.488+0000 7f72527fc640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f723c06be30 0x7f723c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.489+0000 7f72527fc640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f723c06be30 0x7f723c06e2f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f725c06c320 tx=0x7f7244009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:15.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.492+0000 7f7253fff640 1 -- 172.21.15.16:0/20058647 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f724c04b030 con 0x7f725c0af940 2024-04-24T23:50:15.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.744+0000 7f72612cd640 1 -- 172.21.15.16:0/20058647 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7f725c04f5e0 con 0x7f725c0af940 2024-04-24T23:50:15.745 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.746+0000 7f7253fff640 1 -- 172.21.15.16:0/20058647 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7f725c147b90 con 0x7f725c0af940 2024-04-24T23:50:15.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.748+0000 7f72612cd640 1 -- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f723c06be30 msgr2=0x7f723c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:15.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.748+0000 7f72612cd640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f723c06be30 0x7f723c06e2f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f725c06c320 tx=0x7f7244009040 comp rx=0 tx=0).stop 2024-04-24T23:50:15.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.748+0000 7f72612cd640 1 -- 172.21.15.16:0/20058647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f725c0af940 msgr2=0x7f725c13e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:15.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.748+0000 7f72612cd640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f725c0af940 0x7f725c13e740 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f724c009870 tx=0x7f724c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.748+0000 7f72612cd640 1 -- 172.21.15.16:0/20058647 shutdown_connections 2024-04-24T23:50:15.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.748+0000 7f72612cd640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f723c06be30 0x7f723c06e2f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.748+0000 7f72612cd640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f725c1622d0 0x7f725c15eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.748+0000 7f72612cd640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f725c1472f0 0x7f725c15eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.748+0000 7f72612cd640 1 --2- 172.21.15.16:0/20058647 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f725c0af940 0x7f725c13e740 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.748+0000 7f72612cd640 1 -- 172.21.15.16:0/20058647 >> 172.21.15.16:0/20058647 conn(0x7f725c071b40 msgr2=0x7f725c0ceda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:15.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.748+0000 7f72612cd640 1 -- 172.21.15.16:0/20058647 shutdown_connections 2024-04-24T23:50:15.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.748+0000 7f72612cd640 1 -- 172.21.15.16:0/20058647 wait complete. 2024-04-24T23:50:15.758 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:50:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.859+0000 7f3e78031640 1 Processor -- start 2024-04-24T23:50:15.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.859+0000 7f3e78031640 1 -- start start 2024-04-24T23:50:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.860+0000 7f3e78031640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e70144b20 0x7f3e70144f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.860+0000 7f3e78031640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e701457b0 0x7f3e700ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.860+0000 7f3e78031640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3e7015f990 0x7f3e700cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.860+0000 7f3e78031640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f3e70145440 con 0x7f3e70144b20 2024-04-24T23:50:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.860+0000 7f3e78031640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f3e7015fe90 con 0x7f3e7015f990 2024-04-24T23:50:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.860+0000 7f3e78031640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f3e70160170 con 0x7f3e701457b0 2024-04-24T23:50:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.860+0000 7f3e75dcd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e70144b20 0x7f3e70144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.860+0000 7f3e75dcd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e70144b20 0x7f3e70144f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50826/0 (socket says 172.21.15.16:50826) 2024-04-24T23:50:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.860+0000 7f3e75dcd640 1 -- 172.21.15.16:0/2080265148 learned_addr learned my addr 172.21.15.16:0/2080265148 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.861+0000 7f3e755cc640 1 --2- 172.21.15.16:0/2080265148 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e701457b0 0x7f3e700ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.861+0000 7f3e765ce640 1 --2- 172.21.15.16:0/2080265148 >> v2:172.21.15.156:3300/0 conn(0x7f3e7015f990 0x7f3e700cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.861+0000 7f3e75dcd640 1 -- 172.21.15.16:0/2080265148 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e701457b0 msgr2=0x7f3e700ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.861+0000 7f3e75dcd640 1 --2- 172.21.15.16:0/2080265148 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e701457b0 0x7f3e700ce680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.861+0000 7f3e75dcd640 1 -- 172.21.15.16:0/2080265148 >> v2:172.21.15.156:3300/0 conn(0x7f3e7015f990 msgr2=0x7f3e700cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.861+0000 7f3e75dcd640 1 --2- 172.21.15.16:0/2080265148 >> v2:172.21.15.156:3300/0 conn(0x7f3e7015f990 0x7f3e700cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.861+0000 7f3e75dcd640 1 -- 172.21.15.16:0/2080265148 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e70160450 con 0x7f3e70144b20 2024-04-24T23:50:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.861+0000 7f3e75dcd640 1 --2- 172.21.15.16:0/2080265148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e70144b20 0x7f3e70144f00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3e7006eed0 tx=0x7f3e6802a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b0dbb1038b5ebb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:15.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.861+0000 7f3e74dcb640 1 -- 172.21.15.16:0/2080265148 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e68037070 con 0x7f3e70144b20 2024-04-24T23:50:15.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.862+0000 7f3e74dcb640 1 -- 172.21.15.16:0/2080265148 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e68032440 con 0x7f3e70144b20 2024-04-24T23:50:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.862+0000 7f3e74dcb640 1 -- 172.21.15.16:0/2080265148 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e68032810 con 0x7f3e70144b20 2024-04-24T23:50:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.862+0000 7f3e78031640 1 -- 172.21.15.16:0/2080265148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e70144b20 msgr2=0x7f3e70144f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.862+0000 7f3e78031640 1 --2- 172.21.15.16:0/2080265148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e70144b20 0x7f3e70144f00 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3e7006eed0 tx=0x7f3e6802a140 comp rx=0 tx=0).stop 2024-04-24T23:50:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.862+0000 7f3e78031640 1 -- 172.21.15.16:0/2080265148 shutdown_connections 2024-04-24T23:50:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.862+0000 7f3e78031640 1 --2- 172.21.15.16:0/2080265148 >> v2:172.21.15.156:3300/0 conn(0x7f3e7015f990 0x7f3e700cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.862+0000 7f3e78031640 1 --2- 172.21.15.16:0/2080265148 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e701457b0 0x7f3e700ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.862+0000 7f3e78031640 1 --2- 172.21.15.16:0/2080265148 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e70144b20 0x7f3e70144f00 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.862+0000 7f3e78031640 1 -- 172.21.15.16:0/2080265148 >> 172.21.15.16:0/2080265148 conn(0x7f3e700b2000 msgr2=0x7f3e700cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.862+0000 7f3e78031640 1 -- 172.21.15.16:0/2080265148 shutdown_connections 2024-04-24T23:50:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.862+0000 7f3e78031640 1 -- 172.21.15.16:0/2080265148 wait complete. 2024-04-24T23:50:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.863+0000 7f3e78031640 1 Processor -- start 2024-04-24T23:50:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.863+0000 7f3e78031640 1 -- start start 2024-04-24T23:50:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.864+0000 7f3e78031640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e70144b20 0x7f3e701f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.864+0000 7f3e78031640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e701457b0 0x7f3e7013a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.864+0000 7f3e78031640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e7015f990 0x7f3e701f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.864+0000 7f3e78031640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3e700b3390 con 0x7f3e701457b0 2024-04-24T23:50:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.864+0000 7f3e78031640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7006d4b0 con 0x7f3e70144b20 2024-04-24T23:50:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.864+0000 7f3e78031640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f3e70071a00 con 0x7f3e7015f990 2024-04-24T23:50:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.864+0000 7f3e755cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e701457b0 0x7f3e7013a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.864+0000 7f3e755cc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e701457b0 0x7f3e7013a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50840/0 (socket says 172.21.15.16:50840) 2024-04-24T23:50:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.864+0000 7f3e755cc640 1 -- 172.21.15.16:0/2301154681 learned_addr learned my addr 172.21.15.16:0/2301154681 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.864+0000 7f3e75dcd640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e70144b20 0x7f3e701f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.864+0000 7f3e765ce640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e7015f990 0x7f3e701f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.864+0000 7f3e755cc640 1 -- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e7015f990 msgr2=0x7f3e701f1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.865+0000 7f3e755cc640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e7015f990 0x7f3e701f1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.865+0000 7f3e755cc640 1 -- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e70144b20 msgr2=0x7f3e701f5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.865+0000 7f3e755cc640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e70144b20 0x7f3e701f5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.865+0000 7f3e755cc640 1 -- 172.21.15.16:0/2301154681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7013a9e0 con 0x7f3e701457b0 2024-04-24T23:50:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.865+0000 7f3e75dcd640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e70144b20 0x7f3e701f5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.865+0000 7f3e755cc640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e701457b0 0x7f3e7013a4a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000d800 tx=0x7f3e6000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.865+0000 7f3e667fc640 1 -- 172.21.15.16:0/2301154681 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e60018020 con 0x7f3e701457b0 2024-04-24T23:50:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.865+0000 7f3e667fc640 1 -- 172.21.15.16:0/2301154681 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e600096a0 con 0x7f3e701457b0 2024-04-24T23:50:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.865+0000 7f3e78031640 1 -- 172.21.15.16:0/2301154681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e7015b790 con 0x7f3e701457b0 2024-04-24T23:50:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.865+0000 7f3e667fc640 1 -- 172.21.15.16:0/2301154681 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e60009a70 con 0x7f3e701457b0 2024-04-24T23:50:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.865+0000 7f3e78031640 1 -- 172.21.15.16:0/2301154681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e701f5490 con 0x7f3e701457b0 2024-04-24T23:50:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.866+0000 7f3e78031640 1 -- 172.21.15.16:0/2301154681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e700731f0 con 0x7f3e701457b0 2024-04-24T23:50:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.866+0000 7f3e667fc640 1 -- 172.21.15.16:0/2301154681 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3e6001b020 con 0x7f3e701457b0 2024-04-24T23:50:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.867+0000 7f3e667fc640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3e5406be60 0x7f3e5406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.867+0000 7f3e667fc640 1 -- 172.21.15.16:0/2301154681 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f3e6008ae80 con 0x7f3e701457b0 2024-04-24T23:50:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.867+0000 7f3e75dcd640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3e5406be60 0x7f3e5406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.868+0000 7f3e75dcd640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3e5406be60 0x7f3e5406e320 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3e68034000 tx=0x7f3e6802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:15.871+0000 7f3e667fc640 1 -- 172.21.15.16:0/2301154681 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3e60022030 con 0x7f3e701457b0 2024-04-24T23:50:16.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.056+0000 7f3e78031640 1 -- 172.21.15.16:0/2301154681 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3e7004f5e0 con 0x7f3e701457b0 2024-04-24T23:50:16.058 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.058+0000 7f3e667fc640 1 -- 172.21.15.16:0/2301154681 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+13152 (secure 0 0 0) 0x7f3e7013a9e0 con 0x7f3e701457b0 2024-04-24T23:50:16.058 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:16.058 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:50:13.059546+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:50:16.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.061+0000 7f3e78031640 1 -- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3e5406be60 msgr2=0x7f3e5406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.061+0000 7f3e78031640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3e5406be60 0x7f3e5406e320 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3e68034000 tx=0x7f3e6802e000 comp rx=0 tx=0).stop 2024-04-24T23:50:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.061+0000 7f3e78031640 1 -- 172.21.15.16:0/2301154681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e701457b0 msgr2=0x7f3e7013a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.061+0000 7f3e78031640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e701457b0 0x7f3e7013a4a0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f3e6000d800 tx=0x7f3e6000dcc0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.061+0000 7f3e78031640 1 -- 172.21.15.16:0/2301154681 shutdown_connections 2024-04-24T23:50:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.061+0000 7f3e78031640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3e5406be60 0x7f3e5406e320 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.061+0000 7f3e78031640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3e7015f990 0x7f3e701f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.061+0000 7f3e78031640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e701457b0 0x7f3e7013a4a0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.061+0000 7f3e78031640 1 --2- 172.21.15.16:0/2301154681 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3e70144b20 0x7f3e701f5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.061+0000 7f3e78031640 1 -- 172.21.15.16:0/2301154681 >> 172.21.15.16:0/2301154681 conn(0x7f3e700b2000 msgr2=0x7f3e700b4370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.061+0000 7f3e78031640 1 -- 172.21.15.16:0/2301154681 shutdown_connections 2024-04-24T23:50:16.061 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.061+0000 7f3e78031640 1 -- 172.21.15.16:0/2301154681 wait complete. 2024-04-24T23:50:16.072 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T23:50:16.073 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:50:16.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.176+0000 7f9cc3b7e640 1 Processor -- start 2024-04-24T23:50:16.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.177+0000 7f9cc3b7e640 1 -- start start 2024-04-24T23:50:16.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.177+0000 7f9cc3b7e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9cbc06bc40 0x7f9cbc06c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.177+0000 7f9cc3b7e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cbc0d24d0 con 0x7f9cbc0b7050 2024-04-24T23:50:16.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.177+0000 7f9cc3b7e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cbc0d27b0 con 0x7f9cbc15dfd0 2024-04-24T23:50:16.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.177+0000 7f9cc3b7e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f9cbc06ba00 con 0x7f9cbc06bc40 2024-04-24T23:50:16.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc211b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9cbc0b7050 0x7f9cbc0b7460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59066/0 (socket says 172.21.15.16:59066) 2024-04-24T23:50:16.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc211b640 1 -- 172.21.15.16:0/3689624770 learned_addr learned my addr 172.21.15.16:0/3689624770 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:16.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc1119640 1 --2- 172.21.15.16:0/3689624770 >> v2:172.21.15.156:3300/0 conn(0x7f9cbc06bc40 0x7f9cbc06c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 -- 172.21.15.16:0/3689624770 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2614332761 0 0) 0x7f9cbc0d24d0 con 0x7f9cbc0b7050 2024-04-24T23:50:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 -- 172.21.15.16:0/3689624770 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ca4003520 con 0x7f9cbc0b7050 2024-04-24T23:50:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 -- 172.21.15.16:0/3689624770 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2729627373 0 0) 0x7f9ca4003520 con 0x7f9cbc0b7050 2024-04-24T23:50:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 -- 172.21.15.16:0/3689624770 >> v2:172.21.15.156:3300/0 conn(0x7f9cbc06bc40 msgr2=0x7f9cbc06c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 --2- 172.21.15.16:0/3689624770 >> v2:172.21.15.156:3300/0 conn(0x7f9cbc06bc40 0x7f9cbc06c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 -- 172.21.15.16:0/3689624770 >> v1:172.21.15.156:6789/0 conn(0x7f9cbc15dfd0 legacy=0x7f9cbc06b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 -- 172.21.15.16:0/3689624770 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cbc0d2a90 con 0x7f9cbc0b7050 2024-04-24T23:50:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 -- 172.21.15.16:0/3689624770 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f9cac0019f0 con 0x7f9cbc0b7050 2024-04-24T23:50:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 -- 172.21.15.16:0/3689624770 >> v1:172.21.15.16:6789/0 conn(0x7f9cbc0b7050 legacy=0x7f9cbc0b7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 --2- 172.21.15.16:0/3689624770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca40039b0 0x7f9ca4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 --2- 172.21.15.16:0/3689624770 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9ca4004800 0x7f9ca4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 --2- 172.21.15.16:0/3689624770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ca40051c0 0x7f9ca400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 -- 172.21.15.16:0/3689624770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc06ba00 con 0x7f9ca40039b0 2024-04-24T23:50:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 -- 172.21.15.16:0/3689624770 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca400c160 con 0x7f9ca40051c0 2024-04-24T23:50:16.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc0918640 1 -- 172.21.15.16:0/3689624770 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca400c360 con 0x7f9ca4004800 2024-04-24T23:50:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc191a640 1 --2- 172.21.15.16:0/3689624770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca40039b0 0x7f9ca4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc211b640 1 --2- 172.21.15.16:0/3689624770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ca40051c0 0x7f9ca400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.178+0000 7f9cc1119640 1 --2- 172.21.15.16:0/3689624770 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9ca4004800 0x7f9ca4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.179+0000 7f9cc191a640 1 -- 172.21.15.16:0/3689624770 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9ca4004800 msgr2=0x7f9ca4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.179+0000 7f9cc191a640 1 --2- 172.21.15.16:0/3689624770 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9ca4004800 0x7f9ca4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.179+0000 7f9cc191a640 1 -- 172.21.15.16:0/3689624770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ca40051c0 msgr2=0x7f9ca400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.179+0000 7f9cc191a640 1 --2- 172.21.15.16:0/3689624770 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ca40051c0 0x7f9ca400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.179+0000 7f9cc211b640 1 -- 172.21.15.16:0/3689624770 reap_dead start 2024-04-24T23:50:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.179+0000 7f9cc191a640 1 -- 172.21.15.16:0/3689624770 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca4003520 con 0x7f9ca40039b0 2024-04-24T23:50:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.179+0000 7f9cc191a640 1 --2- 172.21.15.16:0/3689624770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca40039b0 0x7f9ca4003dd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4001750 tx=0x7f9ca4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a64da3fbd0149 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.179+0000 7f9cc0918640 1 -- 172.21.15.16:0/3689624770 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb4001370 con 0x7f9ca40039b0 2024-04-24T23:50:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.179+0000 7f9cc0918640 1 -- 172.21.15.16:0/3689624770 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cb40052d0 con 0x7f9ca40039b0 2024-04-24T23:50:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.179+0000 7f9cc0918640 1 -- 172.21.15.16:0/3689624770 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb402abc0 con 0x7f9ca40039b0 2024-04-24T23:50:16.184 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.180+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/3689624770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca40039b0 msgr2=0x7f9ca4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.180+0000 7f9cc3b7e640 1 --2- 172.21.15.16:0/3689624770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca40039b0 0x7f9ca4003dd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4001750 tx=0x7f9ca4005830 comp rx=0 tx=0).stop 2024-04-24T23:50:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.180+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/3689624770 shutdown_connections 2024-04-24T23:50:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.180+0000 7f9cc3b7e640 1 --2- 172.21.15.16:0/3689624770 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca40039b0 0x7f9ca4003dd0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.180+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/3689624770 >> 172.21.15.16:0/3689624770 conn(0x7f9cbc0d3dc0 msgr2=0x7f9cbc0d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.180+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/3689624770 shutdown_connections 2024-04-24T23:50:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.180+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/3689624770 wait complete. 2024-04-24T23:50:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.181+0000 7f9cc3b7e640 1 Processor -- start 2024-04-24T23:50:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.181+0000 7f9cc3b7e640 1 -- start start 2024-04-24T23:50:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc3b7e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca40039b0 0x7f9cbc0cc140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc3b7e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cbc1e76f0 0x7f9cbc15bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc3b7e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc1e7b70 0x7f9cbc15c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc3b7e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc0cead0 con 0x7f9ca40039b0 2024-04-24T23:50:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc3b7e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc0d3050 con 0x7f9cbc1e76f0 2024-04-24T23:50:16.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc3b7e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f9cbc06c0d0 con 0x7f9cbc1e7b70 2024-04-24T23:50:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc191a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca40039b0 0x7f9cbc0cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc191a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca40039b0 0x7f9cbc0cc140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50862/0 (socket says 172.21.15.16:50862) 2024-04-24T23:50:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc191a640 1 -- 172.21.15.16:0/198006749 learned_addr learned my addr 172.21.15.16:0/198006749 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc211b640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc1e7b70 0x7f9cbc15c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc1119640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cbc1e76f0 0x7f9cbc15bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc191a640 1 -- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc1e7b70 msgr2=0x7f9cbc15c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc191a640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc1e7b70 0x7f9cbc15c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc191a640 1 -- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cbc1e76f0 msgr2=0x7f9cbc15bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc191a640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cbc1e76f0 0x7f9cbc15bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc191a640 1 -- 172.21.15.16:0/198006749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cbc0cb0a0 con 0x7f9ca40039b0 2024-04-24T23:50:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.182+0000 7f9cc1119640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cbc1e76f0 0x7f9cbc15bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.183+0000 7f9cc191a640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca40039b0 0x7f9cbc0cc140 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4005040 tx=0x7f9cb4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.183+0000 7f9cba7fc640 1 -- 172.21.15.16:0/198006749 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb4037040 con 0x7f9ca40039b0 2024-04-24T23:50:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.183+0000 7f9cba7fc640 1 -- 172.21.15.16:0/198006749 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cb4040070 con 0x7f9ca40039b0 2024-04-24T23:50:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.183+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/198006749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cbc0cb380 con 0x7f9ca40039b0 2024-04-24T23:50:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.183+0000 7f9cba7fc640 1 -- 172.21.15.16:0/198006749 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb4007a30 con 0x7f9ca40039b0 2024-04-24T23:50:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.183+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/198006749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cbc0cb940 con 0x7f9ca40039b0 2024-04-24T23:50:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.184+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/198006749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cbc068f90 con 0x7f9ca40039b0 2024-04-24T23:50:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.184+0000 7f9cba7fc640 1 -- 172.21.15.16:0/198006749 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9cb40052d0 con 0x7f9ca40039b0 2024-04-24T23:50:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.185+0000 7f9cba7fc640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9ca806be60 0x7f9ca806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.185+0000 7f9cba7fc640 1 -- 172.21.15.16:0/198006749 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f9cb40aa9b0 con 0x7f9ca40039b0 2024-04-24T23:50:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.185+0000 7f9cc1119640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9ca806be60 0x7f9ca806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.186+0000 7f9cc1119640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9ca806be60 0x7f9ca806e320 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc06b840 tx=0x7f9cb0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:16.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.189+0000 7f9cba7fc640 1 -- 172.21.15.16:0/198006749 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9cb4079480 con 0x7f9ca40039b0 2024-04-24T23:50:16.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.371+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/198006749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9cbc06b590 con 0x7f9ca40039b0 2024-04-24T23:50:16.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.371+0000 7f9cba7fc640 1 -- 172.21.15.16:0/198006749 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v35) ==== 93+0+10 (secure 0 0 0) 0x7f9cb4078d80 con 0x7f9ca40039b0 2024-04-24T23:50:16.371 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:50:16.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.373+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9ca806be60 msgr2=0x7f9ca806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.373+0000 7f9cc3b7e640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9ca806be60 0x7f9ca806e320 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9cbc06b840 tx=0x7f9cb0008040 comp rx=0 tx=0).stop 2024-04-24T23:50:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.373+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/198006749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca40039b0 msgr2=0x7f9cbc0cc140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.373+0000 7f9cc3b7e640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca40039b0 0x7f9cbc0cc140 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4005040 tx=0x7f9cb4001120 comp rx=0 tx=0).stop 2024-04-24T23:50:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.373+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/198006749 shutdown_connections 2024-04-24T23:50:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.373+0000 7f9cc3b7e640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9ca806be60 0x7f9ca806e320 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.373+0000 7f9cc3b7e640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc1e7b70 0x7f9cbc15c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.373+0000 7f9cc3b7e640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cbc1e76f0 0x7f9cbc15bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.373+0000 7f9cc3b7e640 1 --2- 172.21.15.16:0/198006749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca40039b0 0x7f9cbc0cc140 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.373+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/198006749 >> 172.21.15.16:0/198006749 conn(0x7f9cbc0d3dc0 msgr2=0x7f9cbc0b8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.373+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/198006749 shutdown_connections 2024-04-24T23:50:16.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.373+0000 7f9cc3b7e640 1 -- 172.21.15.16:0/198006749 wait complete. 2024-04-24T23:50:16.384 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-24T23:50:16.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.485+0000 7f0de91ee640 1 Processor -- start 2024-04-24T23:50:16.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.485+0000 7f0de91ee640 1 -- start start 2024-04-24T23:50:16.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.486+0000 7f0de91ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0de4144740 0x7f0de415c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.486+0000 7f0de91ee640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0de415c8e0 0x7f0de40ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.486+0000 7f0de91ee640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0de40cf3a0 con 0x7f0de4071a00 2024-04-24T23:50:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.486+0000 7f0de91ee640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f0de415c1f0 con 0x7f0de415c8e0 2024-04-24T23:50:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.486+0000 7f0de91ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0de40cc8d0 con 0x7f0de4144740 2024-04-24T23:50:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.486+0000 7f0de359e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0de415c8e0 0x7f0de40ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.486+0000 7f0de359e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0de415c8e0 0x7f0de40ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50872/0 (socket says 172.21.15.16:50872) 2024-04-24T23:50:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.486+0000 7f0de359e640 1 -- 172.21.15.16:0/78645047 learned_addr learned my addr 172.21.15.16:0/78645047 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.486+0000 7f0de259c640 1 --2- 172.21.15.16:0/78645047 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0de4144740 0x7f0de415c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.486+0000 7f0de359e640 1 -- 172.21.15.16:0/78645047 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0de4144740 msgr2=0x7f0de415c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.486+0000 7f0de359e640 1 --2- 172.21.15.16:0/78645047 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0de4144740 0x7f0de415c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.486+0000 7f0de359e640 1 -- 172.21.15.16:0/78645047 >> v1:172.21.15.156:6789/0 conn(0x7f0de4071a00 legacy=0x7f0de4071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.486+0000 7f0de359e640 1 -- 172.21.15.16:0/78645047 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de40ccbb0 con 0x7f0de415c8e0 2024-04-24T23:50:16.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.486+0000 7f0de259c640 1 --2- 172.21.15.16:0/78645047 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0de4144740 0x7f0de415c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:16.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.487+0000 7f0de359e640 1 --2- 172.21.15.16:0/78645047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0de415c8e0 0x7f0de40ce7c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009870 tx=0x7f0dd402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24579f99580015cd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:16.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.487+0000 7f0de1d9b640 1 -- 172.21.15.16:0/78645047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4037070 con 0x7f0de415c8e0 2024-04-24T23:50:16.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.487+0000 7f0de1d9b640 1 -- 172.21.15.16:0/78645047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd4032440 con 0x7f0de415c8e0 2024-04-24T23:50:16.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.487+0000 7f0de1d9b640 1 -- 172.21.15.16:0/78645047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4032810 con 0x7f0de415c8e0 2024-04-24T23:50:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.488+0000 7f0de91ee640 1 -- 172.21.15.16:0/78645047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0de415c8e0 msgr2=0x7f0de40ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.488+0000 7f0de91ee640 1 --2- 172.21.15.16:0/78645047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0de415c8e0 0x7f0de40ce7c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0dd4009870 tx=0x7f0dd402a210 comp rx=0 tx=0).stop 2024-04-24T23:50:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.488+0000 7f0de91ee640 1 -- 172.21.15.16:0/78645047 shutdown_connections 2024-04-24T23:50:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.488+0000 7f0de91ee640 1 --2- 172.21.15.16:0/78645047 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0de4144740 0x7f0de415c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.488+0000 7f0de91ee640 1 --2- 172.21.15.16:0/78645047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0de415c8e0 0x7f0de40ce7c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.488+0000 7f0de91ee640 1 -- 172.21.15.16:0/78645047 >> 172.21.15.16:0/78645047 conn(0x7f0de40b2000 msgr2=0x7f0de40cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.488+0000 7f0de91ee640 1 -- 172.21.15.16:0/78645047 shutdown_connections 2024-04-24T23:50:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.488+0000 7f0de91ee640 1 -- 172.21.15.16:0/78645047 wait complete. 2024-04-24T23:50:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.489+0000 7f0de91ee640 1 Processor -- start 2024-04-24T23:50:16.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.489+0000 7f0de91ee640 1 -- start start 2024-04-24T23:50:16.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de91ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0de4071a00 0x7f0de40b2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de91ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0de415c8e0 0x7f0de40cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de91ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0de40cb7f0 0x7f0de40ac240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de91ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0de40ced00 con 0x7f0de40cb7f0 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de91ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0de40cd170 con 0x7f0de4071a00 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de91ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0de40cee80 con 0x7f0de415c8e0 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0de40cb7f0 0x7f0de40ac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de359e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0de40cb7f0 0x7f0de40ac240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50882/0 (socket says 172.21.15.16:50882) 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de359e640 1 -- 172.21.15.16:0/1167901682 learned_addr learned my addr 172.21.15.16:0/1167901682 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de259c640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0de415c8e0 0x7f0de40cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de2d9d640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0de4071a00 0x7f0de40b2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de359e640 1 -- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0de415c8e0 msgr2=0x7f0de40cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de359e640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0de415c8e0 0x7f0de40cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de359e640 1 -- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0de4071a00 msgr2=0x7f0de40b2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de359e640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0de4071a00 0x7f0de40b2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de359e640 1 -- 172.21.15.16:0/1167901682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0de40cbc90 con 0x7f0de40cb7f0 2024-04-24T23:50:16.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de259c640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0de415c8e0 0x7f0de40cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:16.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.490+0000 7f0de2d9d640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0de4071a00 0x7f0de40b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:16.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.491+0000 7f0de359e640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0de40cb7f0 0x7f0de40ac240 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0dd402a250 tx=0x7f0dd40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:16.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.491+0000 7f0ddb7fe640 1 -- 172.21.15.16:0/1167901682 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4037040 con 0x7f0de40cb7f0 2024-04-24T23:50:16.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.491+0000 7f0ddb7fe640 1 -- 172.21.15.16:0/1167901682 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dd4042070 con 0x7f0de40cb7f0 2024-04-24T23:50:16.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.491+0000 7f0de91ee640 1 -- 172.21.15.16:0/1167901682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0de40c8ff0 con 0x7f0de40cb7f0 2024-04-24T23:50:16.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.491+0000 7f0ddb7fe640 1 -- 172.21.15.16:0/1167901682 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd4030c90 con 0x7f0de40cb7f0 2024-04-24T23:50:16.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.491+0000 7f0de91ee640 1 -- 172.21.15.16:0/1167901682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0de40c95b0 con 0x7f0de40cb7f0 2024-04-24T23:50:16.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.492+0000 7f0de91ee640 1 -- 172.21.15.16:0/1167901682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0de40731f0 con 0x7f0de40cb7f0 2024-04-24T23:50:16.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.492+0000 7f0ddb7fe640 1 -- 172.21.15.16:0/1167901682 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0dd4047460 con 0x7f0de40cb7f0 2024-04-24T23:50:16.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.493+0000 7f0ddb7fe640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0dc406be60 0x7f0dc406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.493+0000 7f0ddb7fe640 1 -- 172.21.15.16:0/1167901682 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7f0dd40ab7e0 con 0x7f0de40cb7f0 2024-04-24T23:50:16.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.493+0000 7f0de2d9d640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0dc406be60 0x7f0dc406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.493+0000 7f0de2d9d640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0dc406be60 0x7f0dc406e320 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0de40b3c00 tx=0x7f0ddc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:16.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.497+0000 7f0ddb7fe640 1 -- 172.21.15.16:0/1167901682 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0dd407a3e0 con 0x7f0de40cb7f0 2024-04-24T23:50:16.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.681+0000 7f0de91ee640 1 -- 172.21.15.16:0/1167901682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f0de40731f0 con 0x7f0de40cb7f0 2024-04-24T23:50:16.680 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.681+0000 7f0ddb7fe640 1 -- 172.21.15.16:0/1167901682 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v35) ==== 84+0+65 (secure 0 0 0) 0x7f0dd4079ce0 con 0x7f0de40cb7f0 2024-04-24T23:50:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:16.681 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2024-04-24T23:50:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: { 2024-04-24T23:50:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2024-04-24T23:50:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": ".mgr" 2024-04-24T23:50:16.681 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:16.681 INFO:teuthology.orchestra.run.smithi016.stdout:] 2024-04-24T23:50:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.683+0000 7f0de91ee640 1 -- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0dc406be60 msgr2=0x7f0dc406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.683+0000 7f0de91ee640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0dc406be60 0x7f0dc406e320 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0de40b3c00 tx=0x7f0ddc009040 comp rx=0 tx=0).stop 2024-04-24T23:50:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.683+0000 7f0de91ee640 1 -- 172.21.15.16:0/1167901682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0de40cb7f0 msgr2=0x7f0de40ac240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.683+0000 7f0de91ee640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0de40cb7f0 0x7f0de40ac240 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f0dd402a250 tx=0x7f0dd40046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.683+0000 7f0de91ee640 1 -- 172.21.15.16:0/1167901682 shutdown_connections 2024-04-24T23:50:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.683+0000 7f0de91ee640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0dc406be60 0x7f0dc406e320 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.683+0000 7f0de91ee640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0de40cb7f0 0x7f0de40ac240 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.683+0000 7f0de91ee640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0de415c8e0 0x7f0de40cb2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.683+0000 7f0de91ee640 1 --2- 172.21.15.16:0/1167901682 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0de4071a00 0x7f0de40b2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.683+0000 7f0de91ee640 1 -- 172.21.15.16:0/1167901682 >> 172.21.15.16:0/1167901682 conn(0x7f0de40b2000 msgr2=0x7f0de41424d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.683+0000 7f0de91ee640 1 -- 172.21.15.16:0/1167901682 shutdown_connections 2024-04-24T23:50:16.683 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.683+0000 7f0de91ee640 1 -- 172.21.15.16:0/1167901682 wait complete. 2024-04-24T23:50:16.693 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-24T23:50:16.694 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-24T23:50:16.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.798+0000 7fde1c2bd640 1 Processor -- start 2024-04-24T23:50:16.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.798+0000 7fde1c2bd640 1 -- start start 2024-04-24T23:50:16.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.799+0000 7fde1c2bd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1415dfb0 0x7fde140cfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.799+0000 7fde1c2bd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde1406c110 0x7fde14162250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.799+0000 7fde1c2bd640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde140d1aa0 con 0x7fde1406bcb0 2024-04-24T23:50:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.799+0000 7fde1c2bd640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fde140d1dd0 con 0x7fde1406c110 2024-04-24T23:50:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.799+0000 7fde1c2bd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fde1406b470 con 0x7fde1415dfb0 2024-04-24T23:50:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.799+0000 7fde1a85a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde1406c110 0x7fde14162250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.799+0000 7fde1a85a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde1406c110 0x7fde14162250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40198/0 (socket says 172.21.15.16:40198) 2024-04-24T23:50:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.799+0000 7fde1a85a640 1 -- 172.21.15.16:0/1172870400 learned_addr learned my addr 172.21.15.16:0/1172870400 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.800+0000 7fde1a059640 1 --2- 172.21.15.16:0/1172870400 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1415dfb0 0x7fde140cfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.800+0000 7fde1a85a640 1 -- 172.21.15.16:0/1172870400 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1415dfb0 msgr2=0x7fde140cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.800+0000 7fde1a85a640 1 --2- 172.21.15.16:0/1172870400 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1415dfb0 0x7fde140cfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.800+0000 7fde1a85a640 1 -- 172.21.15.16:0/1172870400 >> v1:172.21.15.156:6789/0 conn(0x7fde1406bcb0 legacy=0x7fde14161b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.800 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.800+0000 7fde1a85a640 1 -- 172.21.15.16:0/1172870400 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde1406b730 con 0x7fde1406c110 2024-04-24T23:50:16.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.800+0000 7fde1a85a640 1 --2- 172.21.15.16:0/1172870400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1406c110 0x7fde14162250 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fde04009e50 tx=0x7fde0402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca610d0f29bfd36a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:16.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.801+0000 7fde19057640 1 -- 172.21.15.16:0/1172870400 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde04036070 con 0x7fde1406c110 2024-04-24T23:50:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.801+0000 7fde19057640 1 -- 172.21.15.16:0/1172870400 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde04031600 con 0x7fde1406c110 2024-04-24T23:50:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.801+0000 7fde19057640 1 -- 172.21.15.16:0/1172870400 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde040319d0 con 0x7fde1406c110 2024-04-24T23:50:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.801+0000 7fde1c2bd640 1 -- 172.21.15.16:0/1172870400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1406c110 msgr2=0x7fde14162250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.801+0000 7fde1c2bd640 1 --2- 172.21.15.16:0/1172870400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1406c110 0x7fde14162250 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fde04009e50 tx=0x7fde0402a710 comp rx=0 tx=0).stop 2024-04-24T23:50:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.801+0000 7fde1c2bd640 1 -- 172.21.15.16:0/1172870400 shutdown_connections 2024-04-24T23:50:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.801+0000 7fde1c2bd640 1 --2- 172.21.15.16:0/1172870400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1406c110 0x7fde14162250 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.801+0000 7fde1c2bd640 1 --2- 172.21.15.16:0/1172870400 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1415dfb0 0x7fde140cfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.801+0000 7fde1c2bd640 1 -- 172.21.15.16:0/1172870400 >> 172.21.15.16:0/1172870400 conn(0x7fde140d3e30 msgr2=0x7fde140d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.801+0000 7fde1c2bd640 1 -- 172.21.15.16:0/1172870400 shutdown_connections 2024-04-24T23:50:16.802 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.801+0000 7fde1c2bd640 1 -- 172.21.15.16:0/1172870400 wait complete. 2024-04-24T23:50:16.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.802+0000 7fde1c2bd640 1 Processor -- start 2024-04-24T23:50:16.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.802+0000 7fde1c2bd640 1 -- start start 2024-04-24T23:50:16.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.803+0000 7fde1c2bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1406bcb0 0x7fde140ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.803+0000 7fde1c2bd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1406c110 0x7fde14150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.803+0000 7fde1c2bd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde1415dfb0 0x7fde140cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.803+0000 7fde1c2bd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fde1406ba10 con 0x7fde1406bcb0 2024-04-24T23:50:16.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.803+0000 7fde1c2bd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fde1415e390 con 0x7fde1415dfb0 2024-04-24T23:50:16.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.803+0000 7fde1c2bd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fde140b6f60 con 0x7fde1406c110 2024-04-24T23:50:16.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.803+0000 7fde1a059640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1406bcb0 0x7fde140ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.803+0000 7fde1a059640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1406bcb0 0x7fde140ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40204/0 (socket says 172.21.15.16:40204) 2024-04-24T23:50:16.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.803+0000 7fde1a059640 1 -- 172.21.15.16:0/4238717958 learned_addr learned my addr 172.21.15.16:0/4238717958 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:16.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.803+0000 7fde19858640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1406c110 0x7fde14150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.803+0000 7fde1a85a640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde1415dfb0 0x7fde140cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.804+0000 7fde1a059640 1 -- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1406c110 msgr2=0x7fde14150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.804+0000 7fde1a059640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1406c110 0x7fde14150220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.804+0000 7fde1a059640 1 -- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde1415dfb0 msgr2=0x7fde140cd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:16.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.804+0000 7fde1a059640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde1415dfb0 0x7fde140cd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:16.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.804+0000 7fde1a059640 1 -- 172.21.15.16:0/4238717958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde140cd140 con 0x7fde1406bcb0 2024-04-24T23:50:16.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.804+0000 7fde1a85a640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde1415dfb0 0x7fde140cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:16.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.804+0000 7fde1a059640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1406bcb0 0x7fde140ccc00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00e9b0 tx=0x7fde0c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:16.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.804+0000 7fde12ffd640 1 -- 172.21.15.16:0/4238717958 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c017070 con 0x7fde1406bcb0 2024-04-24T23:50:16.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.804+0000 7fde12ffd640 1 -- 172.21.15.16:0/4238717958 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde0c00c9a0 con 0x7fde1406bcb0 2024-04-24T23:50:16.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.804+0000 7fde1c2bd640 1 -- 172.21.15.16:0/4238717958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde140cbb60 con 0x7fde1406bcb0 2024-04-24T23:50:16.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.804+0000 7fde12ffd640 1 -- 172.21.15.16:0/4238717958 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde0c00cd90 con 0x7fde1406bcb0 2024-04-24T23:50:16.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.804+0000 7fde1c2bd640 1 -- 172.21.15.16:0/4238717958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde140cc120 con 0x7fde1406bcb0 2024-04-24T23:50:16.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.805+0000 7fde1c2bd640 1 -- 172.21.15.16:0/4238717958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde14068f90 con 0x7fde1406bcb0 2024-04-24T23:50:16.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.805+0000 7fde12ffd640 1 -- 172.21.15.16:0/4238717958 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fde0c01a070 con 0x7fde1406bcb0 2024-04-24T23:50:16.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.806+0000 7fde12ffd640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fde0006be30 0x7fde0006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:16.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.806+0000 7fde12ffd640 1 -- 172.21.15.16:0/4238717958 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 5249+0+0 (secure 0 0 0) 0x7fde0c08ab40 con 0x7fde1406bcb0 2024-04-24T23:50:16.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.806+0000 7fde19858640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fde0006be30 0x7fde0006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:16.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.806+0000 7fde19858640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fde0006be30 0x7fde0006e2f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fde14150b10 tx=0x7fde08009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:16.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.810+0000 7fde12ffd640 1 -- 172.21.15.16:0/4238717958 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fde0c0590f0 con 0x7fde1406bcb0 2024-04-24T23:50:16.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:16.995+0000 7fde1c2bd640 1 -- 172.21.15.16:0/4238717958 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fde140cecb0 con 0x7fde1406bcb0 2024-04-24T23:50:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.061+0000 7fde12ffd640 1 -- 172.21.15.16:0/4238717958 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v36) ==== 138+0+0 (secure 0 0 0) 0x7fde0c05d3f0 con 0x7fde1406bcb0 2024-04-24T23:50:17.060 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2024-04-24T23:50:17.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.064+0000 7fde1c2bd640 1 -- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fde0006be30 msgr2=0x7fde0006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:17.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.064+0000 7fde1c2bd640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fde0006be30 0x7fde0006e2f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fde14150b10 tx=0x7fde08009040 comp rx=0 tx=0).stop 2024-04-24T23:50:17.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.064+0000 7fde1c2bd640 1 -- 172.21.15.16:0/4238717958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1406bcb0 msgr2=0x7fde140ccc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:17.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.064+0000 7fde1c2bd640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1406bcb0 0x7fde140ccc00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fde0c00e9b0 tx=0x7fde0c00ee70 comp rx=0 tx=0).stop 2024-04-24T23:50:17.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.064+0000 7fde1c2bd640 1 -- 172.21.15.16:0/4238717958 shutdown_connections 2024-04-24T23:50:17.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.064+0000 7fde1c2bd640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fde0006be30 0x7fde0006e2f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:17.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.064+0000 7fde1c2bd640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fde1415dfb0 0x7fde140cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:17.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.064+0000 7fde1c2bd640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fde1406c110 0x7fde14150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:17.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.064+0000 7fde1c2bd640 1 --2- 172.21.15.16:0/4238717958 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde1406bcb0 0x7fde140ccc00 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:17.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.064+0000 7fde1c2bd640 1 -- 172.21.15.16:0/4238717958 >> 172.21.15.16:0/4238717958 conn(0x7fde140d3e30 msgr2=0x7fde14162360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:17.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.065+0000 7fde1c2bd640 1 -- 172.21.15.16:0/4238717958 shutdown_connections 2024-04-24T23:50:17.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.066+0000 7fde1c2bd640 1 -- 172.21.15.16:0/4238717958 wait complete. 2024-04-24T23:50:17.078 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-24T23:50:17.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.187+0000 7f005d9a0640 1 Processor -- start 2024-04-24T23:50:17.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.187+0000 7f005d9a0640 1 -- start start 2024-04-24T23:50:17.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.188+0000 7f005d9a0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f005815df60 0x7f005806ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:17.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.188+0000 7f005d9a0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f00580b2e10 0x7f005806bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:17.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.188+0000 7f005d9a0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f005806c490 0x7f0058161a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:17.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.188+0000 7f005d9a0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f005815e340 con 0x7f00580b2e10 2024-04-24T23:50:17.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.188+0000 7f005d9a0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f00580cf360 con 0x7f005815df60 2024-04-24T23:50:17.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.188+0000 7f005d9a0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f00580cf640 con 0x7f005806c490 2024-04-24T23:50:17.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.188+0000 7f0057fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f00580b2e10 0x7f005806bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:17.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.188+0000 7f0057fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f00580b2e10 0x7f005806bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40220/0 (socket says 172.21.15.16:40220) 2024-04-24T23:50:17.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.188+0000 7f0057fff640 1 -- 172.21.15.16:0/421692613 learned_addr learned my addr 172.21.15.16:0/421692613 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:17.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.189+0000 7f005c99e640 1 --2- 172.21.15.16:0/421692613 >> v2:172.21.15.156:3300/0 conn(0x7f005815df60 0x7f005806ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:17.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.189+0000 7f005d19f640 1 --2- 172.21.15.16:0/421692613 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f005806c490 0x7f0058161a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:17.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.189+0000 7f0057fff640 1 -- 172.21.15.16:0/421692613 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f005806c490 msgr2=0x7f0058161a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:17.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.189+0000 7f0057fff640 1 --2- 172.21.15.16:0/421692613 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f005806c490 0x7f0058161a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:17.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.189+0000 7f0057fff640 1 -- 172.21.15.16:0/421692613 >> v2:172.21.15.156:3300/0 conn(0x7f005815df60 msgr2=0x7f005806ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:17.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.189+0000 7f0057fff640 1 --2- 172.21.15.16:0/421692613 >> v2:172.21.15.156:3300/0 conn(0x7f005815df60 0x7f005806ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:17.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.189+0000 7f0057fff640 1 -- 172.21.15.16:0/421692613 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00580cf920 con 0x7f00580b2e10 2024-04-24T23:50:17.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.189+0000 7f005c99e640 1 --2- 172.21.15.16:0/421692613 >> v2:172.21.15.156:3300/0 conn(0x7f005815df60 0x7f005806ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:17.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.189+0000 7f005d19f640 1 --2- 172.21.15.16:0/421692613 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f005806c490 0x7f0058161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:17.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.189+0000 7f0057fff640 1 --2- 172.21.15.16:0/421692613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00580b2e10 0x7f005806bf50 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f005808c6a0 tx=0x7f004802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3812345787db01 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:17.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.190+0000 7f00577fe640 1 -- 172.21.15.16:0/421692613 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0048037070 con 0x7f00580b2e10 2024-04-24T23:50:17.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.190+0000 7f00577fe640 1 -- 172.21.15.16:0/421692613 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0048032440 con 0x7f00580b2e10 2024-04-24T23:50:17.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.190+0000 7f00577fe640 1 -- 172.21.15.16:0/421692613 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0048032810 con 0x7f00580b2e10 2024-04-24T23:50:17.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.190+0000 7f005d9a0640 1 -- 172.21.15.16:0/421692613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00580b2e10 msgr2=0x7f005806bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:17.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.190+0000 7f005d9a0640 1 --2- 172.21.15.16:0/421692613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00580b2e10 0x7f005806bf50 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f005808c6a0 tx=0x7f004802a1e0 comp rx=0 tx=0).stop 2024-04-24T23:50:17.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.190+0000 7f005d9a0640 1 -- 172.21.15.16:0/421692613 shutdown_connections 2024-04-24T23:50:17.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.190+0000 7f005d9a0640 1 --2- 172.21.15.16:0/421692613 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f005806c490 0x7f0058161a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:17.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.190+0000 7f005d9a0640 1 --2- 172.21.15.16:0/421692613 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00580b2e10 0x7f005806bf50 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:17.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.190+0000 7f005d9a0640 1 --2- 172.21.15.16:0/421692613 >> v2:172.21.15.156:3300/0 conn(0x7f005815df60 0x7f005806ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:17.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.190+0000 7f005d9a0640 1 -- 172.21.15.16:0/421692613 >> 172.21.15.16:0/421692613 conn(0x7f00580d4080 msgr2=0x7f00580d4490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:17.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.190+0000 7f005d9a0640 1 -- 172.21.15.16:0/421692613 shutdown_connections 2024-04-24T23:50:17.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.190+0000 7f005d9a0640 1 -- 172.21.15.16:0/421692613 wait complete. 2024-04-24T23:50:17.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.191+0000 7f005d9a0640 1 Processor -- start 2024-04-24T23:50:17.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.191+0000 7f005d9a0640 1 -- start start 2024-04-24T23:50:17.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.192+0000 7f005d9a0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f005806c490 0x7f00580c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:17.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.192+0000 7f005d9a0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00580b2e10 0x7f00580c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:17.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.192+0000 7f005d9a0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f005815df60 0x7f00580c9a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:17.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.192+0000 7f005d9a0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f00580cf120 con 0x7f005806c490 2024-04-24T23:50:17.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.192+0000 7f005d9a0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f00580cefa0 con 0x7f00580b2e10 2024-04-24T23:50:17.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.192+0000 7f005d9a0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0058161f80 con 0x7f005815df60 2024-04-24T23:50:17.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.192+0000 7f005c99e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f005806c490 0x7f00580c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:17.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.192+0000 7f005c99e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f005806c490 0x7f00580c7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40230/0 (socket says 172.21.15.16:40230) 2024-04-24T23:50:17.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.192+0000 7f005c99e640 1 -- 172.21.15.16:0/289171444 learned_addr learned my addr 172.21.15.16:0/289171444 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:17.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f005d19f640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f005815df60 0x7f00580c9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:17.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f0057fff640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00580b2e10 0x7f00580c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:17.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f005c99e640 1 -- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f005815df60 msgr2=0x7f00580c9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:17.202 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f005c99e640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f005815df60 0x7f00580c9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f005c99e640 1 -- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00580b2e10 msgr2=0x7f00580c9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f005c99e640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00580b2e10 0x7f00580c9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f005c99e640 1 -- 172.21.15.16:0/289171444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00580c7b00 con 0x7f005806c490 2024-04-24T23:50:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f005d19f640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f005815df60 0x7f00580c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f0057fff640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00580b2e10 0x7f00580c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f005c99e640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f005806c490 0x7f00580c7400 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0050007ef0 tx=0x7f005000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f00557fa640 1 -- 172.21.15.16:0/289171444 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f005001a070 con 0x7f005806c490 2024-04-24T23:50:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f00557fa640 1 -- 172.21.15.16:0/289171444 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0050003b90 con 0x7f005806c490 2024-04-24T23:50:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f005d9a0640 1 -- 172.21.15.16:0/289171444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00580c7de0 con 0x7f005806c490 2024-04-24T23:50:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f00557fa640 1 -- 172.21.15.16:0/289171444 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0050004030 con 0x7f005806c490 2024-04-24T23:50:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.193+0000 7f005d9a0640 1 -- 172.21.15.16:0/289171444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00580c83a0 con 0x7f005806c490 2024-04-24T23:50:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.194+0000 7f005d9a0640 1 -- 172.21.15.16:0/289171444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f005804f500 con 0x7f005806c490 2024-04-24T23:50:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.195+0000 7f00557fa640 1 -- 172.21.15.16:0/289171444 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f005001d070 con 0x7f005806c490 2024-04-24T23:50:17.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.195+0000 7f00557fa640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f003806be30 0x7f003806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:17.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.195+0000 7f00557fa640 1 -- 172.21.15.16:0/289171444 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 5619+0+0 (secure 0 0 0) 0x7f005008af60 con 0x7f005806c490 2024-04-24T23:50:17.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.196+0000 7f0057fff640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f003806be30 0x7f003806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:17.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.196+0000 7f0057fff640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f003806be30 0x7f003806e2f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0048034000 tx=0x7f004802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:17.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.200+0000 7f00557fa640 1 -- 172.21.15.16:0/289171444 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0050010070 con 0x7f005806c490 2024-04-24T23:50:17.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:17.384+0000 7f005d9a0640 1 -- 172.21.15.16:0/289171444 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f00580ceac0 con 0x7f005806c490 2024-04-24T23:50:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.066+0000 7f00557fa640 1 -- 172.21.15.16:0/289171444 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v37) ==== 170+0+0 (secure 0 0 0) 0x7f00500593a0 con 0x7f005806c490 2024-04-24T23:50:18.066 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2024-04-24T23:50:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.070+0000 7f005d9a0640 1 -- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f003806be30 msgr2=0x7f003806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.070+0000 7f005d9a0640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f003806be30 0x7f003806e2f0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0048034000 tx=0x7f004802e000 comp rx=0 tx=0).stop 2024-04-24T23:50:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.070+0000 7f005d9a0640 1 -- 172.21.15.16:0/289171444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f005806c490 msgr2=0x7f00580c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.070+0000 7f005d9a0640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f005806c490 0x7f00580c7400 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0050007ef0 tx=0x7f005000d4a0 comp rx=0 tx=0).stop 2024-04-24T23:50:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.070+0000 7f005d9a0640 1 -- 172.21.15.16:0/289171444 shutdown_connections 2024-04-24T23:50:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.070+0000 7f005d9a0640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f003806be30 0x7f003806e2f0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.070+0000 7f005d9a0640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f005815df60 0x7f00580c9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.070+0000 7f005d9a0640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00580b2e10 0x7f00580c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.070+0000 7f005d9a0640 1 --2- 172.21.15.16:0/289171444 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f005806c490 0x7f00580c7400 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.070+0000 7f005d9a0640 1 -- 172.21.15.16:0/289171444 >> 172.21.15.16:0/289171444 conn(0x7f00580d4080 msgr2=0x7f005815d300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:18.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.070+0000 7f005d9a0640 1 -- 172.21.15.16:0/289171444 shutdown_connections 2024-04-24T23:50:18.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.070+0000 7f005d9a0640 1 -- 172.21.15.16:0/289171444 wait complete. 2024-04-24T23:50:18.096 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-24T23:50:18.205 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.206+0000 7ff4ce283640 1 Processor -- start 2024-04-24T23:50:18.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.206+0000 7ff4ce283640 1 -- start start 2024-04-24T23:50:18.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4ce283640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff4c80721a0 0x7ff4c80ce330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4ce283640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c80cf0a0 con 0x7ff4c80ce870 2024-04-24T23:50:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4ce283640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c806bbf0 con 0x7ff4c81444c0 2024-04-24T23:50:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4ce283640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c806be90 con 0x7ff4c80721a0 2024-04-24T23:50:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4cc820640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff4c80ce870 0x7ff4c806cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51662/0 (socket says 172.21.15.16:51662) 2024-04-24T23:50:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4cc820640 1 -- 172.21.15.16:0/149579624 learned_addr learned my addr 172.21.15.16:0/149579624 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c77fe640 1 --2- 172.21.15.16:0/149579624 >> v2:172.21.15.156:3300/0 conn(0x7ff4c80721a0 0x7ff4c80ce330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 -- 172.21.15.16:0/149579624 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1037760948 0 0) 0x7ff4c80cf0a0 con 0x7ff4c80ce870 2024-04-24T23:50:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 -- 172.21.15.16:0/149579624 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4b0005730 con 0x7ff4c80ce870 2024-04-24T23:50:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 -- 172.21.15.16:0/149579624 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1194367320 0 0) 0x7ff4b0005730 con 0x7ff4c80ce870 2024-04-24T23:50:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 -- 172.21.15.16:0/149579624 >> v2:172.21.15.156:3300/0 conn(0x7ff4c80721a0 msgr2=0x7ff4c80ce330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:18.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 --2- 172.21.15.16:0/149579624 >> v2:172.21.15.156:3300/0 conn(0x7ff4c80721a0 0x7ff4c80ce330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 -- 172.21.15.16:0/149579624 >> v1:172.21.15.156:6789/0 conn(0x7ff4c81444c0 legacy=0x7ff4c8071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 -- 172.21.15.16:0/149579624 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c806c170 con 0x7ff4c80ce870 2024-04-24T23:50:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 -- 172.21.15.16:0/149579624 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7ff4c00019f0 con 0x7ff4c80ce870 2024-04-24T23:50:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 -- 172.21.15.16:0/149579624 >> v1:172.21.15.16:6789/0 conn(0x7ff4c80ce870 legacy=0x7ff4c806cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 --2- 172.21.15.16:0/149579624 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4b0005bc0 0x7ff4b0005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 --2- 172.21.15.16:0/149579624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4b0006a10 0x7ff4b0006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 --2- 172.21.15.16:0/149579624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b00073d0 0x7ff4b000dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 -- 172.21.15.16:0/149579624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c806be90 con 0x7ff4b00073d0 2024-04-24T23:50:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 -- 172.21.15.16:0/149579624 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff4b000e270 con 0x7ff4b0006a10 2024-04-24T23:50:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.207+0000 7ff4c6ffd640 1 -- 172.21.15.16:0/149579624 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff4b000e470 con 0x7ff4b0005bc0 2024-04-24T23:50:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.208+0000 7ff4cc820640 1 --2- 172.21.15.16:0/149579624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b00073d0 0x7ff4b000dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:18.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.208+0000 7ff4c77fe640 1 --2- 172.21.15.16:0/149579624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4b0006a10 0x7ff4b0006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.208+0000 7ff4c7fff640 1 --2- 172.21.15.16:0/149579624 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4b0005bc0 0x7ff4b0005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.208+0000 7ff4cc820640 1 -- 172.21.15.16:0/149579624 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4b0005bc0 msgr2=0x7ff4b0005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.208+0000 7ff4cc820640 1 --2- 172.21.15.16:0/149579624 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4b0005bc0 0x7ff4b0005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.208+0000 7ff4cc820640 1 -- 172.21.15.16:0/149579624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4b0006a10 msgr2=0x7ff4b0006e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.208+0000 7ff4cc820640 1 --2- 172.21.15.16:0/149579624 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4b0006a10 0x7ff4b0006e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.208+0000 7ff4cc820640 1 -- 172.21.15.16:0/149579624 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4b0005730 con 0x7ff4b00073d0 2024-04-24T23:50:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.208+0000 7ff4cc820640 1 -- 172.21.15.16:0/149579624 reap_dead start 2024-04-24T23:50:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.209+0000 7ff4cc820640 1 --2- 172.21.15.16:0/149579624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b00073d0 0x7ff4b000dbd0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000a860 tx=0x7ff4b0007a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28b9041072904b57 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.209+0000 7ff4c6ffd640 1 -- 172.21.15.16:0/149579624 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c000a480 con 0x7ff4b00073d0 2024-04-24T23:50:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.209+0000 7ff4c6ffd640 1 -- 172.21.15.16:0/149579624 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4c002f070 con 0x7ff4b00073d0 2024-04-24T23:50:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.209+0000 7ff4c6ffd640 1 -- 172.21.15.16:0/149579624 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c002bca0 con 0x7ff4b00073d0 2024-04-24T23:50:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.209+0000 7ff4ce283640 1 -- 172.21.15.16:0/149579624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b00073d0 msgr2=0x7ff4b000dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.209+0000 7ff4ce283640 1 --2- 172.21.15.16:0/149579624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b00073d0 0x7ff4b000dbd0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000a860 tx=0x7ff4b0007a40 comp rx=0 tx=0).stop 2024-04-24T23:50:18.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.209+0000 7ff4ce283640 1 -- 172.21.15.16:0/149579624 shutdown_connections 2024-04-24T23:50:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.209+0000 7ff4ce283640 1 --2- 172.21.15.16:0/149579624 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b00073d0 0x7ff4b000dbd0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.209+0000 7ff4ce283640 1 -- 172.21.15.16:0/149579624 >> 172.21.15.16:0/149579624 conn(0x7ff4c80cfc50 msgr2=0x7ff4c80d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.209+0000 7ff4ce283640 1 -- 172.21.15.16:0/149579624 shutdown_connections 2024-04-24T23:50:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.209+0000 7ff4ce283640 1 -- 172.21.15.16:0/149579624 wait complete. 2024-04-24T23:50:18.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.210+0000 7ff4ce283640 1 Processor -- start 2024-04-24T23:50:18.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.211+0000 7ff4ce283640 1 -- start start 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.211+0000 7ff4ce283640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b00073d0 0x7ff4c815a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.211+0000 7ff4ce283640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4c81f3a00 0x7ff4c813dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.211+0000 7ff4ce283640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4c80ab8c0 0x7ff4c813f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.211+0000 7ff4ce283640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c81448a0 con 0x7ff4b00073d0 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.211+0000 7ff4ce283640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c806c730 con 0x7ff4c80ab8c0 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.211+0000 7ff4ce283640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c8144a20 con 0x7ff4c81f3a00 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b00073d0 0x7ff4c815a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4c7fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b00073d0 0x7ff4c815a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40244/0 (socket says 172.21.15.16:40244) 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4c7fff640 1 -- 172.21.15.16:0/495481619 learned_addr learned my addr 172.21.15.16:0/495481619 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4c77fe640 1 --2- 172.21.15.16:0/495481619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4c81f3a00 0x7ff4c813dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4cc820640 1 --2- 172.21.15.16:0/495481619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4c80ab8c0 0x7ff4c813f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4c7fff640 1 -- 172.21.15.16:0/495481619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4c81f3a00 msgr2=0x7ff4c813dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4c7fff640 1 --2- 172.21.15.16:0/495481619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4c81f3a00 0x7ff4c813dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4c7fff640 1 -- 172.21.15.16:0/495481619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4c80ab8c0 msgr2=0x7ff4c813f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:18.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4c7fff640 1 --2- 172.21.15.16:0/495481619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4c80ab8c0 0x7ff4c813f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:18.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4c7fff640 1 -- 172.21.15.16:0/495481619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c81f3e80 con 0x7ff4b00073d0 2024-04-24T23:50:18.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4c7fff640 1 --2- 172.21.15.16:0/495481619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b00073d0 0x7ff4c815a0b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800f650 tx=0x7ff4b800fa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:18.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4c4ff9640 1 -- 172.21.15.16:0/495481619 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b80052e0 con 0x7ff4b00073d0 2024-04-24T23:50:18.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4c4ff9640 1 -- 172.21.15.16:0/495481619 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b8018070 con 0x7ff4b00073d0 2024-04-24T23:50:18.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4ce283640 1 -- 172.21.15.16:0/495481619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c80aa820 con 0x7ff4b00073d0 2024-04-24T23:50:18.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4c4ff9640 1 -- 172.21.15.16:0/495481619 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8003810 con 0x7ff4b00073d0 2024-04-24T23:50:18.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.212+0000 7ff4ce283640 1 -- 172.21.15.16:0/495481619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c80aade0 con 0x7ff4b00073d0 2024-04-24T23:50:18.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.213+0000 7ff4ce283640 1 -- 172.21.15.16:0/495481619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4c813e290 con 0x7ff4b00073d0 2024-04-24T23:50:18.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.214+0000 7ff4c4ff9640 1 -- 172.21.15.16:0/495481619 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff4b80039b0 con 0x7ff4b00073d0 2024-04-24T23:50:18.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.214+0000 7ff4c4ff9640 1 --2- 172.21.15.16:0/495481619 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff4b406be30 0x7ff4b406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:18.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.214+0000 7ff4c4ff9640 1 -- 172.21.15.16:0/495481619 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6001+0+0 (secure 0 0 0) 0x7ff4b808ab30 con 0x7ff4b00073d0 2024-04-24T23:50:18.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.215+0000 7ff4c77fe640 1 --2- 172.21.15.16:0/495481619 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff4b406be30 0x7ff4b406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:18.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.215+0000 7ff4c77fe640 1 --2- 172.21.15.16:0/495481619 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff4b406be30 0x7ff4b406e2f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff4c813e4b0 tx=0x7ff4bc009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:18.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.219+0000 7ff4c4ff9640 1 -- 172.21.15.16:0/495481619 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff4b8059440 con 0x7ff4b00073d0 2024-04-24T23:50:18.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:18.440+0000 7ff4ce283640 1 -- 172.21.15.16:0/495481619 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff4c8068f90 con 0x7ff4b00073d0 2024-04-24T23:50:19.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.067+0000 7ff4c4ff9640 1 -- 172.21.15.16:0/495481619 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) ==== 172+0+0 (secure 0 0 0) 0x7ff4b8058d40 con 0x7ff4b00073d0 2024-04-24T23:50:19.067 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 5 and data pool 6 2024-04-24T23:50:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.071+0000 7ff4ce283640 1 -- 172.21.15.16:0/495481619 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff4b406be30 msgr2=0x7ff4b406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.071+0000 7ff4ce283640 1 --2- 172.21.15.16:0/495481619 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff4b406be30 0x7ff4b406e2f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff4c813e4b0 tx=0x7ff4bc009190 comp rx=0 tx=0).stop 2024-04-24T23:50:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.071+0000 7ff4ce283640 1 -- 172.21.15.16:0/495481619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b00073d0 msgr2=0x7ff4c815a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.071+0000 7ff4ce283640 1 --2- 172.21.15.16:0/495481619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b00073d0 0x7ff4c815a0b0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800f650 tx=0x7ff4b800fa50 comp rx=0 tx=0).stop 2024-04-24T23:50:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.072+0000 7ff4ce283640 1 -- 172.21.15.16:0/495481619 shutdown_connections 2024-04-24T23:50:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.072+0000 7ff4ce283640 1 --2- 172.21.15.16:0/495481619 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff4b406be30 0x7ff4b406e2f0 secure :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff4c813e4b0 tx=0x7ff4bc009190 comp rx=0 tx=0).stop 2024-04-24T23:50:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.072+0000 7ff4ce283640 1 --2- 172.21.15.16:0/495481619 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff4c80ab8c0 0x7ff4c813f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.072+0000 7ff4ce283640 1 --2- 172.21.15.16:0/495481619 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff4c81f3a00 0x7ff4c813dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.072+0000 7ff4ce283640 1 --2- 172.21.15.16:0/495481619 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff4b00073d0 0x7ff4c815a0b0 secure :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800f650 tx=0x7ff4b800fa50 comp rx=0 tx=0).stop 2024-04-24T23:50:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.072+0000 7ff4ce283640 1 -- 172.21.15.16:0/495481619 >> 172.21.15.16:0/495481619 conn(0x7ff4c80cfc50 msgr2=0x7ff4c806caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.074+0000 7ff4ce283640 1 -- 172.21.15.16:0/495481619 shutdown_connections 2024-04-24T23:50:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.076+0000 7ff4ce283640 1 -- 172.21.15.16:0/495481619 wait complete. 2024-04-24T23:50:19.103 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:50:19.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.206+0000 7f3a9a3d5640 1 Processor -- start 2024-04-24T23:50:19.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.206+0000 7f3a9a3d5640 1 -- start start 2024-04-24T23:50:19.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.207+0000 7f3a9a3d5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a9415df40 0x7f3a94072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.207+0000 7f3a9a3d5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f3a94162380 0x7f3a940b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.207+0000 7f3a9a3d5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a94162a10 con 0x7f3a940727d0 2024-04-24T23:50:19.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.207+0000 7f3a9a3d5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f3a94072c30 con 0x7f3a94162380 2024-04-24T23:50:19.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.207+0000 7f3a9a3d5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f3a940ae2b0 con 0x7f3a9415df40 2024-04-24T23:50:19.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.207+0000 7f3a937fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3a940727d0 0x7f3a94161c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51670/0 (socket says 172.21.15.16:51670) 2024-04-24T23:50:19.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.207+0000 7f3a937fe640 1 -- 172.21.15.16:0/1547557472 learned_addr learned my addr 172.21.15.16:0/1547557472 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:19.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.207+0000 7f3a93fff640 1 --2- 172.21.15.16:0/1547557472 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a9415df40 0x7f3a94072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.207+0000 7f3a98972640 1 --2- 172.21.15.16:0/1547557472 >> v2:172.21.15.156:3300/0 conn(0x7f3a94162380 0x7f3a940b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.207+0000 7f3a92ffd640 1 -- 172.21.15.16:0/1547557472 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1666996599 0 0) 0x7f3a94162a10 con 0x7f3a940727d0 2024-04-24T23:50:19.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.207+0000 7f3a92ffd640 1 -- 172.21.15.16:0/1547557472 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3a7c003520 con 0x7f3a940727d0 2024-04-24T23:50:19.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.208+0000 7f3a93fff640 1 -- 172.21.15.16:0/1547557472 >> v2:172.21.15.156:3300/0 conn(0x7f3a94162380 msgr2=0x7f3a940b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:19.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.208+0000 7f3a93fff640 1 --2- 172.21.15.16:0/1547557472 >> v2:172.21.15.156:3300/0 conn(0x7f3a94162380 0x7f3a940b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.208+0000 7f3a92ffd640 1 -- 172.21.15.16:0/1547557472 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2139393703 0 0) 0x7f3a7c003520 con 0x7f3a940727d0 2024-04-24T23:50:19.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.208+0000 7f3a93fff640 1 -- 172.21.15.16:0/1547557472 >> v1:172.21.15.16:6789/0 conn(0x7f3a940727d0 legacy=0x7f3a94161c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.208+0000 7f3a93fff640 1 -- 172.21.15.16:0/1547557472 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a940ad980 con 0x7f3a9415df40 2024-04-24T23:50:19.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.208+0000 7f3a93fff640 1 --2- 172.21.15.16:0/1547557472 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a9415df40 0x7f3a94072290 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3a840097a0 tx=0x7f3a8402a110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfcc92f9e05be051 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:19.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.209+0000 7f3a92ffd640 1 -- 172.21.15.16:0/1547557472 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a84037070 con 0x7f3a9415df40 2024-04-24T23:50:19.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.209+0000 7f3a92ffd640 1 -- 172.21.15.16:0/1547557472 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a8402aba0 con 0x7f3a9415df40 2024-04-24T23:50:19.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.209+0000 7f3a92ffd640 1 -- 172.21.15.16:0/1547557472 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a88001670 con 0x7f3a9415df40 2024-04-24T23:50:19.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.209+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/1547557472 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a9415df40 msgr2=0x7f3a94072290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.209+0000 7f3a9a3d5640 1 --2- 172.21.15.16:0/1547557472 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a9415df40 0x7f3a94072290 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3a840097a0 tx=0x7f3a8402a110 comp rx=0 tx=0).stop 2024-04-24T23:50:19.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.209+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/1547557472 shutdown_connections 2024-04-24T23:50:19.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.209+0000 7f3a9a3d5640 1 --2- 172.21.15.16:0/1547557472 >> v2:172.21.15.156:3300/0 conn(0x7f3a94162380 0x7f3a940b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.209+0000 7f3a9a3d5640 1 --2- 172.21.15.16:0/1547557472 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a9415df40 0x7f3a94072290 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.209+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/1547557472 >> 172.21.15.16:0/1547557472 conn(0x7f3a940d0260 msgr2=0x7f3a940d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:19.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.209+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/1547557472 shutdown_connections 2024-04-24T23:50:19.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.209+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/1547557472 wait complete. 2024-04-24T23:50:19.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.210+0000 7f3a9a3d5640 1 Processor -- start 2024-04-24T23:50:19.211 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.211+0000 7f3a9a3d5640 1 -- start start 2024-04-24T23:50:19.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.211+0000 7f3a9a3d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a940727d0 0x7f3a94155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.211+0000 7f3a9a3d5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a9415df40 0x7f3a9415a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.211+0000 7f3a9a3d5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a94162380 0x7f3a9415a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.211+0000 7f3a9a3d5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f3a940ae740 con 0x7f3a940727d0 2024-04-24T23:50:19.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.211+0000 7f3a9a3d5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3a940adf40 con 0x7f3a9415df40 2024-04-24T23:50:19.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.211+0000 7f3a9a3d5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f3a9415e320 con 0x7f3a94162380 2024-04-24T23:50:19.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.211+0000 7f3a93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a940727d0 0x7f3a94155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.211+0000 7f3a93fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a940727d0 0x7f3a94155c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40260/0 (socket says 172.21.15.16:40260) 2024-04-24T23:50:19.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.211+0000 7f3a93fff640 1 -- 172.21.15.16:0/3152215022 learned_addr learned my addr 172.21.15.16:0/3152215022 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:19.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.211+0000 7f3a937fe640 1 --2- 172.21.15.16:0/3152215022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a9415df40 0x7f3a9415a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.211+0000 7f3a98972640 1 --2- 172.21.15.16:0/3152215022 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a94162380 0x7f3a9415a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.212+0000 7f3a93fff640 1 -- 172.21.15.16:0/3152215022 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a94162380 msgr2=0x7f3a9415a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.212+0000 7f3a93fff640 1 --2- 172.21.15.16:0/3152215022 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a94162380 0x7f3a9415a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.212+0000 7f3a93fff640 1 -- 172.21.15.16:0/3152215022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a9415df40 msgr2=0x7f3a9415a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.212+0000 7f3a93fff640 1 --2- 172.21.15.16:0/3152215022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a9415df40 0x7f3a9415a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.212+0000 7f3a93fff640 1 -- 172.21.15.16:0/3152215022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a941561c0 con 0x7f3a940727d0 2024-04-24T23:50:19.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.212+0000 7f3a93fff640 1 --2- 172.21.15.16:0/3152215022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a940727d0 0x7f3a94155c80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3a840097a0 tx=0x7f3a840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:19.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.212+0000 7f3a90ff9640 1 -- 172.21.15.16:0/3152215022 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a84037050 con 0x7f3a940727d0 2024-04-24T23:50:19.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.212+0000 7f3a90ff9640 1 -- 172.21.15.16:0/3152215022 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a84043070 con 0x7f3a940727d0 2024-04-24T23:50:19.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.212+0000 7f3a90ff9640 1 -- 172.21.15.16:0/3152215022 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a84006e10 con 0x7f3a940727d0 2024-04-24T23:50:19.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.212+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/3152215022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a94145c40 con 0x7f3a940727d0 2024-04-24T23:50:19.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.212+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/3152215022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a94156870 con 0x7f3a940727d0 2024-04-24T23:50:19.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.213+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/3152215022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a940afa40 con 0x7f3a940727d0 2024-04-24T23:50:19.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.214+0000 7f3a90ff9640 1 -- 172.21.15.16:0/3152215022 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3a8403ca10 con 0x7f3a940727d0 2024-04-24T23:50:19.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.214+0000 7f3a90ff9640 1 --2- 172.21.15.16:0/3152215022 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a8006be30 0x7f3a8006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.214+0000 7f3a90ff9640 1 -- 172.21.15.16:0/3152215022 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f3a840ab880 con 0x7f3a940727d0 2024-04-24T23:50:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.214+0000 7f3a937fe640 1 --2- 172.21.15.16:0/3152215022 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a8006be30 0x7f3a8006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.215+0000 7f3a937fe640 1 --2- 172.21.15.16:0/3152215022 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a8006be30 0x7f3a8006e2f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3a94156540 tx=0x7f3a880057f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:19.218 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.218+0000 7f3a90ff9640 1 -- 172.21.15.16:0/3152215022 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f3a940afa40 con 0x7f3a940727d0 2024-04-24T23:50:19.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.401+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/3152215022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3a9406e5c0 con 0x7f3a940727d0 2024-04-24T23:50:19.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.402+0000 7f3a90ff9640 1 -- 172.21.15.16:0/3152215022 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16943 (secure 0 0 0) 0x7f3a84079c00 con 0x7f3a940727d0 2024-04-24T23:50:19.402 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:19.402 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":38,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:50:18.442003+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:50:17.026573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-24T23:50:17.415963+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:50:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.404+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/3152215022 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a8006be30 msgr2=0x7f3a8006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.404+0000 7f3a9a3d5640 1 --2- 172.21.15.16:0/3152215022 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a8006be30 0x7f3a8006e2f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3a94156540 tx=0x7f3a880057f0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.404+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/3152215022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a940727d0 msgr2=0x7f3a94155c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.404+0000 7f3a9a3d5640 1 --2- 172.21.15.16:0/3152215022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a940727d0 0x7f3a94155c80 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f3a840097a0 tx=0x7f3a840046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.404+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/3152215022 shutdown_connections 2024-04-24T23:50:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.404+0000 7f3a9a3d5640 1 --2- 172.21.15.16:0/3152215022 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f3a8006be30 0x7f3a8006e2f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.404+0000 7f3a9a3d5640 1 --2- 172.21.15.16:0/3152215022 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f3a94162380 0x7f3a9415a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.404+0000 7f3a9a3d5640 1 --2- 172.21.15.16:0/3152215022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a9415df40 0x7f3a9415a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.404+0000 7f3a9a3d5640 1 --2- 172.21.15.16:0/3152215022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a940727d0 0x7f3a94155c80 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.404+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/3152215022 >> 172.21.15.16:0/3152215022 conn(0x7f3a940d0260 msgr2=0x7f3a940afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:19.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.404+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/3152215022 shutdown_connections 2024-04-24T23:50:19.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.404+0000 7f3a9a3d5640 1 -- 172.21.15.16:0/3152215022 wait complete. 2024-04-24T23:50:19.415 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:50:19.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.515+0000 7f2e52831640 1 Processor -- start 2024-04-24T23:50:19.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.515+0000 7f2e52831640 1 -- start start 2024-04-24T23:50:19.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.516+0000 7f2e52831640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2e4c144740 0x7f2e4c15c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.516+0000 7f2e52831640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e4c15c8e0 0x7f2e4c0ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.516+0000 7f2e52831640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e4c0cf3a0 con 0x7f2e4c071a00 2024-04-24T23:50:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.516+0000 7f2e52831640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f2e4c15c1f0 con 0x7f2e4c144740 2024-04-24T23:50:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.516+0000 7f2e52831640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2e4c0cc8d0 con 0x7f2e4c15c8e0 2024-04-24T23:50:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.517+0000 7f2e50dce640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e4c15c8e0 0x7f2e4c0ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.517+0000 7f2e4b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2e4c144740 0x7f2e4c15c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.517+0000 7f2e50dce640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e4c15c8e0 0x7f2e4c0ce7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:59650/0 (socket says 172.21.15.16:59650) 2024-04-24T23:50:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.517+0000 7f2e50dce640 1 -- 172.21.15.16:0/986320997 learned_addr learned my addr 172.21.15.16:0/986320997 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.517+0000 7f2e50dce640 1 -- 172.21.15.16:0/986320997 >> v2:172.21.15.156:3300/0 conn(0x7f2e4c144740 msgr2=0x7f2e4c15c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.517+0000 7f2e50dce640 1 --2- 172.21.15.16:0/986320997 >> v2:172.21.15.156:3300/0 conn(0x7f2e4c144740 0x7f2e4c15c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.517+0000 7f2e50dce640 1 -- 172.21.15.16:0/986320997 >> v1:172.21.15.156:6789/0 conn(0x7f2e4c071a00 legacy=0x7f2e4c071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.517+0000 7f2e50dce640 1 -- 172.21.15.16:0/986320997 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e4c0ccbb0 con 0x7f2e4c15c8e0 2024-04-24T23:50:19.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.517+0000 7f2e4b7fe640 1 --2- 172.21.15.16:0/986320997 >> v2:172.21.15.156:3300/0 conn(0x7f2e4c144740 0x7f2e4c15c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:50:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.518+0000 7f2e50dce640 1 --2- 172.21.15.16:0/986320997 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e4c15c8e0 0x7f2e4c0ce7c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2e44009e50 tx=0x7f2e4402a730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e28d9e57f50c39c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.518+0000 7f2e4affd640 1 -- 172.21.15.16:0/986320997 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e44036070 con 0x7f2e4c15c8e0 2024-04-24T23:50:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.518+0000 7f2e4affd640 1 -- 172.21.15.16:0/986320997 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e44031600 con 0x7f2e4c15c8e0 2024-04-24T23:50:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.518+0000 7f2e4affd640 1 -- 172.21.15.16:0/986320997 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e440319d0 con 0x7f2e4c15c8e0 2024-04-24T23:50:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.518+0000 7f2e52831640 1 -- 172.21.15.16:0/986320997 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e4c15c8e0 msgr2=0x7f2e4c0ce7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.518+0000 7f2e52831640 1 --2- 172.21.15.16:0/986320997 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e4c15c8e0 0x7f2e4c0ce7c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2e44009e50 tx=0x7f2e4402a730 comp rx=0 tx=0).stop 2024-04-24T23:50:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.518+0000 7f2e52831640 1 -- 172.21.15.16:0/986320997 shutdown_connections 2024-04-24T23:50:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.518+0000 7f2e52831640 1 --2- 172.21.15.16:0/986320997 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e4c15c8e0 0x7f2e4c0ce7c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.518+0000 7f2e52831640 1 --2- 172.21.15.16:0/986320997 >> v2:172.21.15.156:3300/0 conn(0x7f2e4c144740 0x7f2e4c15c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.518+0000 7f2e52831640 1 -- 172.21.15.16:0/986320997 >> 172.21.15.16:0/986320997 conn(0x7f2e4c0b2000 msgr2=0x7f2e4c0cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:19.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.519+0000 7f2e52831640 1 -- 172.21.15.16:0/986320997 shutdown_connections 2024-04-24T23:50:19.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.519+0000 7f2e52831640 1 -- 172.21.15.16:0/986320997 wait complete. 2024-04-24T23:50:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.520+0000 7f2e52831640 1 Processor -- start 2024-04-24T23:50:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.520+0000 7f2e52831640 1 -- start start 2024-04-24T23:50:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e52831640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e4c071a00 0x7f2e4c13dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e52831640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e4c15c8e0 0x7f2e4c0b0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e52831640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c159e60 0x7f2e4c0b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e52831640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4c0ced00 con 0x7f2e4c159e60 2024-04-24T23:50:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e52831640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2e4c0cd170 con 0x7f2e4c15c8e0 2024-04-24T23:50:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e52831640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2e4c0cee80 con 0x7f2e4c071a00 2024-04-24T23:50:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e50dce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c159e60 0x7f2e4c0b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e50dce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c159e60 0x7f2e4c0b10a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40262/0 (socket says 172.21.15.16:40262) 2024-04-24T23:50:19.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e50dce640 1 -- 172.21.15.16:0/2464101961 learned_addr learned my addr 172.21.15.16:0/2464101961 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e4bfff640 1 --2- 172.21.15.16:0/2464101961 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e4c071a00 0x7f2e4c13dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e4b7fe640 1 --2- 172.21.15.16:0/2464101961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e4c15c8e0 0x7f2e4c0b0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e50dce640 1 -- 172.21.15.16:0/2464101961 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e4c071a00 msgr2=0x7f2e4c13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e50dce640 1 --2- 172.21.15.16:0/2464101961 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e4c071a00 0x7f2e4c13dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e50dce640 1 -- 172.21.15.16:0/2464101961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e4c15c8e0 msgr2=0x7f2e4c0b0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e50dce640 1 --2- 172.21.15.16:0/2464101961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e4c15c8e0 0x7f2e4c0b0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e50dce640 1 -- 172.21.15.16:0/2464101961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e4c13cc10 con 0x7f2e4c159e60 2024-04-24T23:50:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e50dce640 1 --2- 172.21.15.16:0/2464101961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c159e60 0x7f2e4c0b10a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f2e44034040 tx=0x7f2e44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e48ff9640 1 -- 172.21.15.16:0/2464101961 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e44041020 con 0x7f2e4c159e60 2024-04-24T23:50:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e48ff9640 1 -- 172.21.15.16:0/2464101961 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e440077a0 con 0x7f2e4c159e60 2024-04-24T23:50:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.521+0000 7f2e48ff9640 1 -- 172.21.15.16:0/2464101961 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e44036040 con 0x7f2e4c159e60 2024-04-24T23:50:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.522+0000 7f2e52831640 1 -- 172.21.15.16:0/2464101961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e4c13ceb0 con 0x7f2e4c159e60 2024-04-24T23:50:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.522+0000 7f2e52831640 1 -- 172.21.15.16:0/2464101961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e4c13d750 con 0x7f2e4c159e60 2024-04-24T23:50:19.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.522+0000 7f2e52831640 1 -- 172.21.15.16:0/2464101961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e4c0731f0 con 0x7f2e4c159e60 2024-04-24T23:50:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.523+0000 7f2e48ff9640 1 -- 172.21.15.16:0/2464101961 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2e44044020 con 0x7f2e4c159e60 2024-04-24T23:50:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.523+0000 7f2e48ff9640 1 --2- 172.21.15.16:0/2464101961 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e3806be30 0x7f2e3806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.523+0000 7f2e48ff9640 1 -- 172.21.15.16:0/2464101961 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f2e440ab9a0 con 0x7f2e4c159e60 2024-04-24T23:50:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.524+0000 7f2e4bfff640 1 --2- 172.21.15.16:0/2464101961 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e3806be30 0x7f2e3806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.525 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.524+0000 7f2e4bfff640 1 --2- 172.21.15.16:0/2464101961 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e3806be30 0x7f2e3806e2f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c1e7df0 tx=0x7f2e3c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:19.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.528+0000 7f2e48ff9640 1 -- 172.21.15.16:0/2464101961 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2e4404b030 con 0x7f2e4c159e60 2024-04-24T23:50:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.712+0000 7f2e52831640 1 -- 172.21.15.16:0/2464101961 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2e4c1412a0 con 0x7f2e4c159e60 2024-04-24T23:50:19.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.713+0000 7f2e48ff9640 1 -- 172.21.15.16:0/2464101961 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) ==== 74+0+16943 (secure 0 0 0) 0x7f2e44079c00 con 0x7f2e4c159e60 2024-04-24T23:50:19.715 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:19.715 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":38,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:50:18.442003+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:50:17.026573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-24T23:50:17.415963+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:50:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.715+0000 7f2e52831640 1 -- 172.21.15.16:0/2464101961 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e3806be30 msgr2=0x7f2e3806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.715+0000 7f2e52831640 1 --2- 172.21.15.16:0/2464101961 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e3806be30 0x7f2e3806e2f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c1e7df0 tx=0x7f2e3c009040 comp rx=0 tx=0).stop 2024-04-24T23:50:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.715+0000 7f2e52831640 1 -- 172.21.15.16:0/2464101961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c159e60 msgr2=0x7f2e4c0b10a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.715+0000 7f2e52831640 1 --2- 172.21.15.16:0/2464101961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c159e60 0x7f2e4c0b10a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f2e44034040 tx=0x7f2e44004a40 comp rx=0 tx=0).stop 2024-04-24T23:50:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.715+0000 7f2e52831640 1 -- 172.21.15.16:0/2464101961 shutdown_connections 2024-04-24T23:50:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.715+0000 7f2e52831640 1 --2- 172.21.15.16:0/2464101961 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e3806be30 0x7f2e3806e2f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.715+0000 7f2e52831640 1 --2- 172.21.15.16:0/2464101961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e4c159e60 0x7f2e4c0b10a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.715+0000 7f2e52831640 1 --2- 172.21.15.16:0/2464101961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e4c15c8e0 0x7f2e4c0b0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.715+0000 7f2e52831640 1 --2- 172.21.15.16:0/2464101961 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2e4c071a00 0x7f2e4c13dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.715+0000 7f2e52831640 1 -- 172.21.15.16:0/2464101961 >> 172.21.15.16:0/2464101961 conn(0x7f2e4c0b2000 msgr2=0x7f2e4c0cd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.716+0000 7f2e52831640 1 -- 172.21.15.16:0/2464101961 shutdown_connections 2024-04-24T23:50:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.716+0000 7f2e52831640 1 -- 172.21.15.16:0/2464101961 wait complete. 2024-04-24T23:50:19.727 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-24T23:50:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.830+0000 7f2b43210640 1 Processor -- start 2024-04-24T23:50:19.830 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.830+0000 7f2b43210640 1 -- start start 2024-04-24T23:50:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.831+0000 7f2b43210640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b3c06bc40 0x7f2b3c06c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.831+0000 7f2b43210640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b3c0d24d0 con 0x7f2b3c0b7050 2024-04-24T23:50:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.831+0000 7f2b43210640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b3c0d27b0 con 0x7f2b3c15dfd0 2024-04-24T23:50:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.831+0000 7f2b43210640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2b3c06c5e0 con 0x7f2b3c06bc40 2024-04-24T23:50:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.831+0000 7f2b417ad640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2b3c0b7050 0x7f2b3c0b74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51674/0 (socket says 172.21.15.16:51674) 2024-04-24T23:50:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.831+0000 7f2b417ad640 1 -- 172.21.15.16:0/2447507684 learned_addr learned my addr 172.21.15.16:0/2447507684 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.832+0000 7f2b3bfff640 1 --2- 172.21.15.16:0/2447507684 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b3c06bc40 0x7f2b3c06c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.832+0000 7f2b3b7fe640 1 -- 172.21.15.16:0/2447507684 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1899163335 0 0) 0x7f2b3c0d24d0 con 0x7f2b3c0b7050 2024-04-24T23:50:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.832+0000 7f2b3b7fe640 1 -- 172.21.15.16:0/2447507684 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2b24003520 con 0x7f2b3c0b7050 2024-04-24T23:50:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.832+0000 7f2b3b7fe640 1 -- 172.21.15.16:0/2447507684 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2188860572 0 0) 0x7f2b3c0d27b0 con 0x7f2b3c15dfd0 2024-04-24T23:50:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.832+0000 7f2b3bfff640 1 -- 172.21.15.16:0/2447507684 >> v1:172.21.15.156:6789/0 conn(0x7f2b3c15dfd0 legacy=0x7f2b3c06b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.832 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.832+0000 7f2b3bfff640 1 -- 172.21.15.16:0/2447507684 >> v1:172.21.15.16:6789/0 conn(0x7f2b3c0b7050 legacy=0x7f2b3c0b74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.832+0000 7f2b3bfff640 1 -- 172.21.15.16:0/2447507684 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b3c0d2a90 con 0x7f2b3c06bc40 2024-04-24T23:50:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.833+0000 7f2b3bfff640 1 --2- 172.21.15.16:0/2447507684 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b3c06bc40 0x7f2b3c06c0a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2b300097d0 tx=0x7f2b3002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f7f1710a38abfe0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.833+0000 7f2b3b7fe640 1 -- 172.21.15.16:0/2447507684 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b30037070 con 0x7f2b3c06bc40 2024-04-24T23:50:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.833+0000 7f2b3b7fe640 1 -- 172.21.15.16:0/2447507684 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b30032440 con 0x7f2b3c06bc40 2024-04-24T23:50:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.833+0000 7f2b3b7fe640 1 -- 172.21.15.16:0/2447507684 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b300327f0 con 0x7f2b3c06bc40 2024-04-24T23:50:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.833+0000 7f2b43210640 1 -- 172.21.15.16:0/2447507684 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b3c06bc40 msgr2=0x7f2b3c06c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.833+0000 7f2b43210640 1 --2- 172.21.15.16:0/2447507684 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b3c06bc40 0x7f2b3c06c0a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2b300097d0 tx=0x7f2b3002a180 comp rx=0 tx=0).stop 2024-04-24T23:50:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.833+0000 7f2b43210640 1 -- 172.21.15.16:0/2447507684 shutdown_connections 2024-04-24T23:50:19.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.833+0000 7f2b43210640 1 --2- 172.21.15.16:0/2447507684 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b3c06bc40 0x7f2b3c06c0a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.833+0000 7f2b43210640 1 -- 172.21.15.16:0/2447507684 >> 172.21.15.16:0/2447507684 conn(0x7f2b3c0d3dc0 msgr2=0x7f2b3c0d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.833+0000 7f2b43210640 1 -- 172.21.15.16:0/2447507684 shutdown_connections 2024-04-24T23:50:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.833+0000 7f2b43210640 1 -- 172.21.15.16:0/2447507684 wait complete. 2024-04-24T23:50:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.834+0000 7f2b43210640 1 Processor -- start 2024-04-24T23:50:19.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.835+0000 7f2b43210640 1 -- start start 2024-04-24T23:50:19.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.835+0000 7f2b43210640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b3c06bc40 0x7f2b3c0cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.835+0000 7f2b43210640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b3c0b7050 0x7f2b3c14f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.835+0000 7f2b43210640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b3c15dfd0 0x7f2b3c15f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.835+0000 7f2b43210640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3c0d3050 con 0x7f2b3c15dfd0 2024-04-24T23:50:19.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.835+0000 7f2b43210640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2b3c06ba00 con 0x7f2b3c0b7050 2024-04-24T23:50:19.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.835+0000 7f2b43210640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2b3c0cead0 con 0x7f2b3c06bc40 2024-04-24T23:50:19.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.835+0000 7f2b417ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b3c15dfd0 0x7f2b3c15f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b417ad640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b3c15dfd0 0x7f2b3c15f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40272/0 (socket says 172.21.15.16:40272) 2024-04-24T23:50:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b417ad640 1 -- 172.21.15.16:0/1332225383 learned_addr learned my addr 172.21.15.16:0/1332225383 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b3bfff640 1 --2- 172.21.15.16:0/1332225383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b3c0b7050 0x7f2b3c14f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b40fac640 1 --2- 172.21.15.16:0/1332225383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b3c06bc40 0x7f2b3c0cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b417ad640 1 -- 172.21.15.16:0/1332225383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b3c06bc40 msgr2=0x7f2b3c0cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b417ad640 1 --2- 172.21.15.16:0/1332225383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b3c06bc40 0x7f2b3c0cd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b417ad640 1 -- 172.21.15.16:0/1332225383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b3c0b7050 msgr2=0x7f2b3c14f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b417ad640 1 --2- 172.21.15.16:0/1332225383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b3c0b7050 0x7f2b3c14f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b417ad640 1 -- 172.21.15.16:0/1332225383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b3c15f900 con 0x7f2b3c15dfd0 2024-04-24T23:50:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b417ad640 1 --2- 172.21.15.16:0/1332225383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b3c15dfd0 0x7f2b3c15f3c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00c460 tx=0x7f2b2c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b397fa640 1 -- 172.21.15.16:0/1332225383 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c017070 con 0x7f2b3c15dfd0 2024-04-24T23:50:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b397fa640 1 -- 172.21.15.16:0/1332225383 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b2c005ce0 con 0x7f2b3c15dfd0 2024-04-24T23:50:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b43210640 1 -- 172.21.15.16:0/1332225383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b3c15e600 con 0x7f2b3c15dfd0 2024-04-24T23:50:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b397fa640 1 -- 172.21.15.16:0/1332225383 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c007310 con 0x7f2b3c15dfd0 2024-04-24T23:50:19.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.836+0000 7f2b43210640 1 -- 172.21.15.16:0/1332225383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b3c15ebc0 con 0x7f2b3c15dfd0 2024-04-24T23:50:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.837+0000 7f2b43210640 1 -- 172.21.15.16:0/1332225383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b3c068f90 con 0x7f2b3c15dfd0 2024-04-24T23:50:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.838+0000 7f2b397fa640 1 -- 172.21.15.16:0/1332225383 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2b2c01a070 con 0x7f2b3c15dfd0 2024-04-24T23:50:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.838+0000 7f2b397fa640 1 --2- 172.21.15.16:0/1332225383 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2b2806be30 0x7f2b2806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.838+0000 7f2b397fa640 1 -- 172.21.15.16:0/1332225383 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 6101+0+0 (secure 0 0 0) 0x7f2b2c08a370 con 0x7f2b3c15dfd0 2024-04-24T23:50:19.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.838+0000 7f2b40fac640 1 --2- 172.21.15.16:0/1332225383 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2b2806be30 0x7f2b2806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:19.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.839+0000 7f2b40fac640 1 --2- 172.21.15.16:0/1332225383 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2b2806be30 0x7f2b2806e2f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2b340012e0 tx=0x7f2b34009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:19.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:19.842+0000 7f2b397fa640 1 -- 172.21.15.16:0/1332225383 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2b2c058540 con 0x7f2b3c15dfd0 2024-04-24T23:50:20.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.062+0000 7f2b43210640 1 -- 172.21.15.16:0/1332225383 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f2b3c0cd4b0 con 0x7f2b3c15dfd0 2024-04-24T23:50:20.087 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.086+0000 7f2b397fa640 1 -- 172.21.15.16:0/1332225383 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) ==== 118+0+0 (secure 0 0 0) 0x7f2b2c05c840 con 0x7f2b3c15dfd0 2024-04-24T23:50:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.088+0000 7f2b43210640 1 -- 172.21.15.16:0/1332225383 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2b2806be30 msgr2=0x7f2b2806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.088+0000 7f2b43210640 1 --2- 172.21.15.16:0/1332225383 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2b2806be30 0x7f2b2806e2f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2b340012e0 tx=0x7f2b34009040 comp rx=0 tx=0).stop 2024-04-24T23:50:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.088+0000 7f2b43210640 1 -- 172.21.15.16:0/1332225383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b3c15dfd0 msgr2=0x7f2b3c15f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.088+0000 7f2b43210640 1 --2- 172.21.15.16:0/1332225383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b3c15dfd0 0x7f2b3c15f3c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c00c460 tx=0x7f2b2c00c920 comp rx=0 tx=0).stop 2024-04-24T23:50:20.089 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.089+0000 7f2b43210640 1 -- 172.21.15.16:0/1332225383 shutdown_connections 2024-04-24T23:50:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.089+0000 7f2b43210640 1 --2- 172.21.15.16:0/1332225383 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2b2806be30 0x7f2b2806e2f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.089+0000 7f2b43210640 1 --2- 172.21.15.16:0/1332225383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b3c15dfd0 0x7f2b3c15f3c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.089+0000 7f2b43210640 1 --2- 172.21.15.16:0/1332225383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2b3c0b7050 0x7f2b3c14f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.089+0000 7f2b43210640 1 --2- 172.21.15.16:0/1332225383 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2b3c06bc40 0x7f2b3c0cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.089+0000 7f2b43210640 1 -- 172.21.15.16:0/1332225383 >> 172.21.15.16:0/1332225383 conn(0x7f2b3c0d3dc0 msgr2=0x7f2b3c0b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.089+0000 7f2b43210640 1 -- 172.21.15.16:0/1332225383 shutdown_connections 2024-04-24T23:50:20.090 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.089+0000 7f2b43210640 1 -- 172.21.15.16:0/1332225383 wait complete. 2024-04-24T23:50:20.099 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-04-24T23:50:20.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-04-24T23:50:20.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.204+0000 7f5d27da9640 1 Processor -- start 2024-04-24T23:50:20.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.204+0000 7f5d27da9640 1 -- start start 2024-04-24T23:50:20.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.205+0000 7f5d27da9640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d2015e2f0 0x7f5d2006b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.205+0000 7f5d27da9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d200b75c0 0x7f5d200b7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.205+0000 7f5d27da9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d2015e6d0 con 0x7f5d200d26e0 2024-04-24T23:50:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.205+0000 7f5d27da9640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f5d200d2ca0 con 0x7f5d2015e2f0 2024-04-24T23:50:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.205+0000 7f5d27da9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f5d200d2f80 con 0x7f5d200b75c0 2024-04-24T23:50:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.205+0000 7f5d25b45640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5d2015e2f0 0x7f5d2006b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.205+0000 7f5d25344640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5d200d26e0 0x7f5d2006b950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51676/0 (socket says 172.21.15.16:51676) 2024-04-24T23:50:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.205+0000 7f5d25344640 1 -- 172.21.15.16:0/4115225877 learned_addr learned my addr 172.21.15.16:0/4115225877 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.205+0000 7f5d26346640 1 --2- 172.21.15.16:0/4115225877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d200b75c0 0x7f5d200b7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.205+0000 7f5d25b45640 1 -- 172.21.15.16:0/4115225877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d200b75c0 msgr2=0x7f5d200b7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.205+0000 7f5d25b45640 1 --2- 172.21.15.16:0/4115225877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d200b75c0 0x7f5d200b7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.205+0000 7f5d25b45640 1 -- 172.21.15.16:0/4115225877 >> v1:172.21.15.16:6789/0 conn(0x7f5d200d26e0 legacy=0x7f5d2006b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.205+0000 7f5d25b45640 1 -- 172.21.15.16:0/4115225877 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d200d3260 con 0x7f5d2015e2f0 2024-04-24T23:50:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.205+0000 7f5d24b43640 1 -- 172.21.15.16:0/4115225877 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1439024172 0 0) 0x7f5d2015e6d0 con 0x7f5d200d26e0 2024-04-24T23:50:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.206+0000 7f5d25b45640 1 --2- 172.21.15.16:0/4115225877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d2015e2f0 0x7f5d2006b410 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c0097a0 tx=0x7f5d1c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b4e4ddab698535 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.206+0000 7f5d24b43640 1 -- 172.21.15.16:0/4115225877 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c037070 con 0x7f5d2015e2f0 2024-04-24T23:50:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.206+0000 7f5d24b43640 1 -- 172.21.15.16:0/4115225877 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d1c032440 con 0x7f5d2015e2f0 2024-04-24T23:50:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.206+0000 7f5d24b43640 1 -- 172.21.15.16:0/4115225877 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c032810 con 0x7f5d2015e2f0 2024-04-24T23:50:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.206+0000 7f5d27da9640 1 -- 172.21.15.16:0/4115225877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d2015e2f0 msgr2=0x7f5d2006b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.206+0000 7f5d27da9640 1 --2- 172.21.15.16:0/4115225877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d2015e2f0 0x7f5d2006b410 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c0097a0 tx=0x7f5d1c02a180 comp rx=0 tx=0).stop 2024-04-24T23:50:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.206+0000 7f5d27da9640 1 -- 172.21.15.16:0/4115225877 shutdown_connections 2024-04-24T23:50:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.206+0000 7f5d27da9640 1 --2- 172.21.15.16:0/4115225877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d200b75c0 0x7f5d200b7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.206+0000 7f5d27da9640 1 --2- 172.21.15.16:0/4115225877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d2015e2f0 0x7f5d2006b410 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.206+0000 7f5d27da9640 1 -- 172.21.15.16:0/4115225877 >> 172.21.15.16:0/4115225877 conn(0x7f5d200d4850 msgr2=0x7f5d200d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.207+0000 7f5d27da9640 1 -- 172.21.15.16:0/4115225877 shutdown_connections 2024-04-24T23:50:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.207+0000 7f5d27da9640 1 -- 172.21.15.16:0/4115225877 wait complete. 2024-04-24T23:50:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.208+0000 7f5d27da9640 1 Processor -- start 2024-04-24T23:50:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.208+0000 7f5d27da9640 1 -- start start 2024-04-24T23:50:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.208+0000 7f5d27da9640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d200b75c0 0x7f5d2014cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.208+0000 7f5d27da9640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d200d26e0 0x7f5d2014f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.208+0000 7f5d27da9640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d2015e2f0 0x7f5d201f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.208+0000 7f5d27da9640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f5d200d0080 con 0x7f5d2015e2f0 2024-04-24T23:50:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.208+0000 7f5d27da9640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f5d200b70a0 con 0x7f5d200d26e0 2024-04-24T23:50:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.208+0000 7f5d27da9640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f5d2006c060 con 0x7f5d200b75c0 2024-04-24T23:50:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d26346640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d2015e2f0 0x7f5d201f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d26346640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d2015e2f0 0x7f5d201f3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40290/0 (socket says 172.21.15.16:40290) 2024-04-24T23:50:20.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d26346640 1 -- 172.21.15.16:0/1301886241 learned_addr learned my addr 172.21.15.16:0/1301886241 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d25b45640 1 --2- 172.21.15.16:0/1301886241 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d200b75c0 0x7f5d2014cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d25344640 1 --2- 172.21.15.16:0/1301886241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d200d26e0 0x7f5d2014f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d26346640 1 -- 172.21.15.16:0/1301886241 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d200b75c0 msgr2=0x7f5d2014cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d26346640 1 --2- 172.21.15.16:0/1301886241 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d200b75c0 0x7f5d2014cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d26346640 1 -- 172.21.15.16:0/1301886241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d200d26e0 msgr2=0x7f5d2014f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d26346640 1 --2- 172.21.15.16:0/1301886241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d200d26e0 0x7f5d2014f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d26346640 1 -- 172.21.15.16:0/1301886241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d2014fd30 con 0x7f5d2015e2f0 2024-04-24T23:50:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d26346640 1 --2- 172.21.15.16:0/1301886241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d2015e2f0 0x7f5d201f3140 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5d18007e40 tx=0x7f5d1800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d167fc640 1 -- 172.21.15.16:0/1301886241 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d18014620 con 0x7f5d2015e2f0 2024-04-24T23:50:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d167fc640 1 -- 172.21.15.16:0/1301886241 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d18019070 con 0x7f5d2015e2f0 2024-04-24T23:50:20.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d27da9640 1 -- 172.21.15.16:0/1301886241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d2014d390 con 0x7f5d2015e2f0 2024-04-24T23:50:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d167fc640 1 -- 172.21.15.16:0/1301886241 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d18014a20 con 0x7f5d2015e2f0 2024-04-24T23:50:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.209+0000 7f5d27da9640 1 -- 172.21.15.16:0/1301886241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d2014dbd0 con 0x7f5d2015e2f0 2024-04-24T23:50:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.210+0000 7f5d27da9640 1 -- 172.21.15.16:0/1301886241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d20068f90 con 0x7f5d2015e2f0 2024-04-24T23:50:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.211+0000 7f5d167fc640 1 -- 172.21.15.16:0/1301886241 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5d18014bc0 con 0x7f5d2015e2f0 2024-04-24T23:50:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.211+0000 7f5d167fc640 1 --2- 172.21.15.16:0/1301886241 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5d0406be30 0x7f5d0406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.211+0000 7f5d167fc640 1 -- 172.21.15.16:0/1301886241 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f5d1808b7b0 con 0x7f5d2015e2f0 2024-04-24T23:50:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.211+0000 7f5d25b45640 1 --2- 172.21.15.16:0/1301886241 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5d0406be30 0x7f5d0406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.212+0000 7f5d25b45640 1 --2- 172.21.15.16:0/1301886241 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5d0406be30 0x7f5d0406e2f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c034000 tx=0x7f5d1c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:20.215 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.216+0000 7f5d167fc640 1 -- 172.21.15.16:0/1301886241 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f5d2014d390 con 0x7f5d2015e2f0 2024-04-24T23:50:20.440 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:20.440+0000 7f5d27da9640 1 -- 172.21.15.16:0/1301886241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f5d2014d670 con 0x7f5d2015e2f0 2024-04-24T23:50:21.091 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.091+0000 7f5d167fc640 1 -- 172.21.15.16:0/1301886241 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) ==== 122+0+0 (secure 0 0 0) 0x7f5d2014fd30 con 0x7f5d2015e2f0 2024-04-24T23:50:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.093+0000 7f5d27da9640 1 -- 172.21.15.16:0/1301886241 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5d0406be30 msgr2=0x7f5d0406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.093+0000 7f5d27da9640 1 --2- 172.21.15.16:0/1301886241 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5d0406be30 0x7f5d0406e2f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c034000 tx=0x7f5d1c02e000 comp rx=0 tx=0).stop 2024-04-24T23:50:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.094+0000 7f5d27da9640 1 -- 172.21.15.16:0/1301886241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d2015e2f0 msgr2=0x7f5d201f3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.094 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.094+0000 7f5d27da9640 1 --2- 172.21.15.16:0/1301886241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d2015e2f0 0x7f5d201f3140 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f5d18007e40 tx=0x7f5d1800e4b0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.094+0000 7f5d27da9640 1 -- 172.21.15.16:0/1301886241 shutdown_connections 2024-04-24T23:50:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.094+0000 7f5d27da9640 1 --2- 172.21.15.16:0/1301886241 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f5d0406be30 0x7f5d0406e2f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.094+0000 7f5d27da9640 1 --2- 172.21.15.16:0/1301886241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5d2015e2f0 0x7f5d201f3140 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.094+0000 7f5d27da9640 1 --2- 172.21.15.16:0/1301886241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f5d200d26e0 0x7f5d2014f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.094+0000 7f5d27da9640 1 --2- 172.21.15.16:0/1301886241 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f5d200b75c0 0x7f5d2014cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.094+0000 7f5d27da9640 1 -- 172.21.15.16:0/1301886241 >> 172.21.15.16:0/1301886241 conn(0x7f5d200d4850 msgr2=0x7f5d200b4aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.094+0000 7f5d27da9640 1 -- 172.21.15.16:0/1301886241 shutdown_connections 2024-04-24T23:50:21.095 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.094+0000 7f5d27da9640 1 -- 172.21.15.16:0/1301886241 wait complete. 2024-04-24T23:50:21.107 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-04-24T23:50:21.107 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:50:21.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.222+0000 7eff21122640 1 Processor -- start 2024-04-24T23:50:21.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.222+0000 7eff21122640 1 -- start start 2024-04-24T23:50:21.223 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.223+0000 7eff21122640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff1c15dfa0 0x7eff1c0d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.223+0000 7eff21122640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7eff1c0ce5f0 0x7eff1c0cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.223+0000 7eff21122640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff1c0d05a0 con 0x7eff1c0b5f00 2024-04-24T23:50:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.223+0000 7eff21122640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7eff1c0b7360 con 0x7eff1c15dfa0 2024-04-24T23:50:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.223+0000 7eff21122640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7eff1c0cc640 con 0x7eff1c0ce5f0 2024-04-24T23:50:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.223+0000 7eff1ad9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7eff1c15dfa0 0x7eff1c0d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.223+0000 7eff1b59e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7eff1c0b5f00 0x7eff1c0b6370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51690/0 (socket says 172.21.15.16:51690) 2024-04-24T23:50:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.223+0000 7eff1b59e640 1 -- 172.21.15.16:0/310336521 learned_addr learned my addr 172.21.15.16:0/310336521 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.223+0000 7eff1ad9d640 1 -- 172.21.15.16:0/310336521 >> v2:172.21.15.156:3300/0 conn(0x7eff1c0ce5f0 msgr2=0x7eff1c0cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.223+0000 7eff1ad9d640 1 --2- 172.21.15.16:0/310336521 >> v2:172.21.15.156:3300/0 conn(0x7eff1c0ce5f0 0x7eff1c0cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.224 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.224+0000 7eff1ad9d640 1 -- 172.21.15.16:0/310336521 >> v1:172.21.15.16:6789/0 conn(0x7eff1c0b5f00 legacy=0x7eff1c0b6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.224+0000 7eff19d9b640 1 -- 172.21.15.16:0/310336521 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1398686963 0 0) 0x7eff1c0d05a0 con 0x7eff1c0b5f00 2024-04-24T23:50:21.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.224+0000 7eff1ad9d640 1 -- 172.21.15.16:0/310336521 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1c0d0880 con 0x7eff1c15dfa0 2024-04-24T23:50:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.224+0000 7eff1ad9d640 1 --2- 172.21.15.16:0/310336521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1c15dfa0 0x7eff1c0d0fe0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7eff1c161a40 tx=0x7eff1402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd8018e093f80ae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.224+0000 7eff19d9b640 1 -- 172.21.15.16:0/310336521 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14037070 con 0x7eff1c15dfa0 2024-04-24T23:50:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.224+0000 7eff19d9b640 1 -- 172.21.15.16:0/310336521 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff14032440 con 0x7eff1c15dfa0 2024-04-24T23:50:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.224+0000 7eff19d9b640 1 -- 172.21.15.16:0/310336521 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14032810 con 0x7eff1c15dfa0 2024-04-24T23:50:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.225+0000 7eff21122640 1 -- 172.21.15.16:0/310336521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1c15dfa0 msgr2=0x7eff1c0d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.225+0000 7eff21122640 1 --2- 172.21.15.16:0/310336521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1c15dfa0 0x7eff1c0d0fe0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7eff1c161a40 tx=0x7eff1402a1a0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.225+0000 7eff21122640 1 -- 172.21.15.16:0/310336521 shutdown_connections 2024-04-24T23:50:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.225+0000 7eff21122640 1 --2- 172.21.15.16:0/310336521 >> v2:172.21.15.156:3300/0 conn(0x7eff1c0ce5f0 0x7eff1c0cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.225+0000 7eff21122640 1 --2- 172.21.15.16:0/310336521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1c15dfa0 0x7eff1c0d0fe0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.225+0000 7eff21122640 1 -- 172.21.15.16:0/310336521 >> 172.21.15.16:0/310336521 conn(0x7eff1c06b4d0 msgr2=0x7eff1c0b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.225+0000 7eff21122640 1 -- 172.21.15.16:0/310336521 shutdown_connections 2024-04-24T23:50:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.225+0000 7eff21122640 1 -- 172.21.15.16:0/310336521 wait complete. 2024-04-24T23:50:21.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.227+0000 7eff21122640 1 Processor -- start 2024-04-24T23:50:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.227+0000 7eff21122640 1 -- start start 2024-04-24T23:50:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.227+0000 7eff21122640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1c0b5f00 0x7eff1c1f66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.227+0000 7eff21122640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff1c0ce5f0 0x7eff1c147b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.227+0000 7eff21122640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff1c15dfa0 0x7eff1c0c7440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.227+0000 7eff21122640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c15e380 con 0x7eff1c0b5f00 2024-04-24T23:50:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.227+0000 7eff21122640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7eff1c0cc2d0 con 0x7eff1c15dfa0 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.227+0000 7eff21122640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7eff1c0d1520 con 0x7eff1c0ce5f0 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.227+0000 7eff1ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1c0b5f00 0x7eff1c1f66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.227+0000 7eff1ad9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1c0b5f00 0x7eff1c1f66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40312/0 (socket says 172.21.15.16:40312) 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.227+0000 7eff1ad9d640 1 -- 172.21.15.16:0/2541515488 learned_addr learned my addr 172.21.15.16:0/2541515488 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.228+0000 7eff1b59e640 1 --2- 172.21.15.16:0/2541515488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff1c15dfa0 0x7eff1c0c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.228+0000 7eff1a59c640 1 --2- 172.21.15.16:0/2541515488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff1c0ce5f0 0x7eff1c147b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.228+0000 7eff1ad9d640 1 -- 172.21.15.16:0/2541515488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff1c0ce5f0 msgr2=0x7eff1c147b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.228+0000 7eff1ad9d640 1 --2- 172.21.15.16:0/2541515488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff1c0ce5f0 0x7eff1c147b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.228+0000 7eff1ad9d640 1 -- 172.21.15.16:0/2541515488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff1c15dfa0 msgr2=0x7eff1c0c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.228+0000 7eff1ad9d640 1 --2- 172.21.15.16:0/2541515488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff1c15dfa0 0x7eff1c0c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.228+0000 7eff1ad9d640 1 -- 172.21.15.16:0/2541515488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff1c0ca3c0 con 0x7eff1c0b5f00 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.228+0000 7eff1ad9d640 1 --2- 172.21.15.16:0/2541515488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1c0b5f00 0x7eff1c1f66f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7eff14009890 tx=0x7eff140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.228+0000 7eff137fe640 1 -- 172.21.15.16:0/2541515488 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14041070 con 0x7eff1c0b5f00 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.228+0000 7eff137fe640 1 -- 172.21.15.16:0/2541515488 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff1403c890 con 0x7eff1c0b5f00 2024-04-24T23:50:21.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.228+0000 7eff21122640 1 -- 172.21.15.16:0/2541515488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff1c0ca6a0 con 0x7eff1c0b5f00 2024-04-24T23:50:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.228+0000 7eff137fe640 1 -- 172.21.15.16:0/2541515488 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14037040 con 0x7eff1c0b5f00 2024-04-24T23:50:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.228+0000 7eff21122640 1 -- 172.21.15.16:0/2541515488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff1c0cac60 con 0x7eff1c0b5f00 2024-04-24T23:50:21.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.229+0000 7eff21122640 1 -- 172.21.15.16:0/2541515488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff1c06bea0 con 0x7eff1c0b5f00 2024-04-24T23:50:21.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.230+0000 7eff137fe640 1 -- 172.21.15.16:0/2541515488 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7eff14044070 con 0x7eff1c0b5f00 2024-04-24T23:50:21.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.230+0000 7eff137fe640 1 --2- 172.21.15.16:0/2541515488 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efefc06be30 0x7efefc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.230+0000 7eff137fe640 1 -- 172.21.15.16:0/2541515488 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7eff140abb30 con 0x7eff1c0b5f00 2024-04-24T23:50:21.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.230+0000 7eff1a59c640 1 --2- 172.21.15.16:0/2541515488 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efefc06be30 0x7efefc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.231+0000 7eff1a59c640 1 --2- 172.21.15.16:0/2541515488 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efefc06be30 0x7efefc06e2f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7eff1c1571e0 tx=0x7eff04009110 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:21.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.235+0000 7eff137fe640 1 -- 172.21.15.16:0/2541515488 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7eff14079eb0 con 0x7eff1c0b5f00 2024-04-24T23:50:21.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.455+0000 7eff21122640 1 -- 172.21.15.16:0/2541515488 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff1c04f5e0 con 0x7eff1c0b5f00 2024-04-24T23:50:21.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.456+0000 7eff137fe640 1 -- 172.21.15.16:0/2541515488 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5181 (secure 0 0 0) 0x7eff1407e1b0 con 0x7eff1c0b5f00 2024-04-24T23:50:21.456 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:21.457 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5299,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/448535992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":448535992},{"type":"v1","addr":"172.21.15.16:6835","nonce":448535992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5305,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1890687780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1890687780},{"type":"v1","addr":"172.21.15.156:6836","nonce":1890687780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5308,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2538147395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":2538147395},{"type":"v1","addr":"172.21.15.156:6837","nonce":2538147395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:50:18.442310+0000","modified":"2024-04-24T23:50:20.441592+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6834/561619268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":561619268},{"type":"v1","addr":"172.21.15.16:6834","nonce":561619268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5490,"qdb_cluster":[5490]},"id":2}]} 2024-04-24T23:50:21.457 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 15 2024-04-24T23:50:21.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.458+0000 7eff21122640 1 -- 172.21.15.16:0/2541515488 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efefc06be30 msgr2=0x7efefc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.458+0000 7eff21122640 1 --2- 172.21.15.16:0/2541515488 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efefc06be30 0x7efefc06e2f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7eff1c1571e0 tx=0x7eff04009110 comp rx=0 tx=0).stop 2024-04-24T23:50:21.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.458+0000 7eff21122640 1 -- 172.21.15.16:0/2541515488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1c0b5f00 msgr2=0x7eff1c1f66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.458 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.458+0000 7eff21122640 1 --2- 172.21.15.16:0/2541515488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1c0b5f00 0x7eff1c1f66f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7eff14009890 tx=0x7eff140046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.458+0000 7eff21122640 1 -- 172.21.15.16:0/2541515488 shutdown_connections 2024-04-24T23:50:21.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.458+0000 7eff21122640 1 --2- 172.21.15.16:0/2541515488 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7efefc06be30 0x7efefc06e2f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.458+0000 7eff21122640 1 --2- 172.21.15.16:0/2541515488 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7eff1c15dfa0 0x7eff1c0c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.458+0000 7eff21122640 1 --2- 172.21.15.16:0/2541515488 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7eff1c0ce5f0 0x7eff1c147b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.458+0000 7eff21122640 1 --2- 172.21.15.16:0/2541515488 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7eff1c0b5f00 0x7eff1c1f66f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.458+0000 7eff21122640 1 -- 172.21.15.16:0/2541515488 >> 172.21.15.16:0/2541515488 conn(0x7eff1c06b4d0 msgr2=0x7eff1c160350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:21.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.458+0000 7eff21122640 1 -- 172.21.15.16:0/2541515488 shutdown_connections 2024-04-24T23:50:21.459 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.458+0000 7eff21122640 1 -- 172.21.15.16:0/2541515488 wait complete. 2024-04-24T23:50:21.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:50:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.571+0000 7f2aba942640 1 Processor -- start 2024-04-24T23:50:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.571+0000 7f2aba942640 1 -- start start 2024-04-24T23:50:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.572+0000 7f2aba942640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ab4072110 0x7f2ab4161a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.572+0000 7f2aba942640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2ab40d0530 0x7f2ab4161fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.572+0000 7f2aba942640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ab40b6a70 con 0x7f2ab415df20 2024-04-24T23:50:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.572+0000 7f2aba942640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f2ab40cf460 con 0x7f2ab4072110 2024-04-24T23:50:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.572+0000 7f2aba942640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f2ab40b6820 con 0x7f2ab40d0530 2024-04-24T23:50:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.572+0000 7f2ab37fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ab4072110 0x7f2ab4161a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.572+0000 7f2ab3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2ab415df20 0x7f2ab4071a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51704/0 (socket says 172.21.15.16:51704) 2024-04-24T23:50:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.572+0000 7f2ab3fff640 1 -- 172.21.15.16:0/1492448307 learned_addr learned my addr 172.21.15.16:0/1492448307 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.572+0000 7f2ab8edf640 1 --2- 172.21.15.16:0/1492448307 >> v2:172.21.15.156:3300/0 conn(0x7f2ab40d0530 0x7f2ab4161fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.572+0000 7f2ab37fe640 1 -- 172.21.15.16:0/1492448307 >> v2:172.21.15.156:3300/0 conn(0x7f2ab40d0530 msgr2=0x7f2ab4161fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.572+0000 7f2ab37fe640 1 --2- 172.21.15.16:0/1492448307 >> v2:172.21.15.156:3300/0 conn(0x7f2ab40d0530 0x7f2ab4161fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.572+0000 7f2ab37fe640 1 -- 172.21.15.16:0/1492448307 >> v1:172.21.15.16:6789/0 conn(0x7f2ab415df20 legacy=0x7f2ab4071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.572+0000 7f2ab37fe640 1 -- 172.21.15.16:0/1492448307 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab40ce6a0 con 0x7f2ab4072110 2024-04-24T23:50:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.572+0000 7f2ab2ffd640 1 -- 172.21.15.16:0/1492448307 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2043943052 0 0) 0x7f2ab40b6a70 con 0x7f2ab415df20 2024-04-24T23:50:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.573+0000 7f2ab37fe640 1 --2- 172.21.15.16:0/1492448307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab4072110 0x7f2ab4161a60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40cd500 tx=0x7f2aa802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e416449ad38ea041 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.573+0000 7f2ab2ffd640 1 -- 172.21.15.16:0/1492448307 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8037070 con 0x7f2ab4072110 2024-04-24T23:50:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.573+0000 7f2ab2ffd640 1 -- 172.21.15.16:0/1492448307 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aa8032440 con 0x7f2ab4072110 2024-04-24T23:50:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.573+0000 7f2ab2ffd640 1 -- 172.21.15.16:0/1492448307 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8032810 con 0x7f2ab4072110 2024-04-24T23:50:21.577 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.573+0000 7f2aba942640 1 -- 172.21.15.16:0/1492448307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab4072110 msgr2=0x7f2ab4161a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.573+0000 7f2aba942640 1 --2- 172.21.15.16:0/1492448307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab4072110 0x7f2ab4161a60 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2ab40cd500 tx=0x7f2aa802a1a0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.574+0000 7f2aba942640 1 -- 172.21.15.16:0/1492448307 shutdown_connections 2024-04-24T23:50:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.574+0000 7f2aba942640 1 --2- 172.21.15.16:0/1492448307 >> v2:172.21.15.156:3300/0 conn(0x7f2ab40d0530 0x7f2ab4161fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.574+0000 7f2aba942640 1 --2- 172.21.15.16:0/1492448307 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab4072110 0x7f2ab4161a60 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.574+0000 7f2aba942640 1 -- 172.21.15.16:0/1492448307 >> 172.21.15.16:0/1492448307 conn(0x7f2ab40d0ac0 msgr2=0x7f2ab40d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.574+0000 7f2aba942640 1 -- 172.21.15.16:0/1492448307 shutdown_connections 2024-04-24T23:50:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.574+0000 7f2aba942640 1 -- 172.21.15.16:0/1492448307 wait complete. 2024-04-24T23:50:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.575+0000 7f2aba942640 1 Processor -- start 2024-04-24T23:50:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.575+0000 7f2aba942640 1 -- start start 2024-04-24T23:50:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2aba942640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab4072110 0x7f2ab414e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2aba942640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab40d0530 0x7f2ab414d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2aba942640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab415df20 0x7f2ab415fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2aba942640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab40cef40 con 0x7f2ab40d0530 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2aba942640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab40cec60 con 0x7f2ab4072110 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2aba942640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab40b4030 con 0x7f2ab415df20 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2ab37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab40d0530 0x7f2ab414d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2ab37fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab40d0530 0x7f2ab414d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40324/0 (socket says 172.21.15.16:40324) 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2ab37fe640 1 -- 172.21.15.16:0/3298721789 learned_addr learned my addr 172.21.15.16:0/3298721789 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2ab37fe640 1 -- 172.21.15.16:0/3298721789 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab415df20 msgr2=0x7f2ab415fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2ab37fe640 1 --2- 172.21.15.16:0/3298721789 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab415df20 0x7f2ab415fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2ab37fe640 1 -- 172.21.15.16:0/3298721789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab4072110 msgr2=0x7f2ab414e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2ab3fff640 1 --2- 172.21.15.16:0/3298721789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab4072110 0x7f2ab414e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2ab37fe640 1 --2- 172.21.15.16:0/3298721789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab4072110 0x7f2ab414e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2ab37fe640 1 -- 172.21.15.16:0/3298721789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab415e880 con 0x7f2ab40d0530 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2ab37fe640 1 --2- 172.21.15.16:0/3298721789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab40d0530 0x7f2ab414d940 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8000c00 tx=0x7f2aa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2ab0ff9640 1 -- 172.21.15.16:0/3298721789 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa8037040 con 0x7f2ab40d0530 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2ab0ff9640 1 -- 172.21.15.16:0/3298721789 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aa8042070 con 0x7f2ab40d0530 2024-04-24T23:50:21.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2aba942640 1 -- 172.21.15.16:0/3298721789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab415eb60 con 0x7f2ab40d0530 2024-04-24T23:50:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2ab0ff9640 1 -- 172.21.15.16:0/3298721789 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa803b950 con 0x7f2ab40d0530 2024-04-24T23:50:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.576+0000 7f2aba942640 1 -- 172.21.15.16:0/3298721789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab415f120 con 0x7f2ab40d0530 2024-04-24T23:50:21.580 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.578+0000 7f2aba942640 1 -- 172.21.15.16:0/3298721789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab415b270 con 0x7f2ab40d0530 2024-04-24T23:50:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.578+0000 7f2ab0ff9640 1 -- 172.21.15.16:0/3298721789 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2aa803baf0 con 0x7f2ab40d0530 2024-04-24T23:50:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.578+0000 7f2ab0ff9640 1 --2- 172.21.15.16:0/3298721789 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2aa006be60 0x7f2aa006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.578+0000 7f2ab0ff9640 1 -- 172.21.15.16:0/3298721789 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f2aa80ab860 con 0x7f2ab40d0530 2024-04-24T23:50:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.579+0000 7f2ab3fff640 1 --2- 172.21.15.16:0/3298721789 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2aa006be60 0x7f2aa006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.581 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.579+0000 7f2ab3fff640 1 --2- 172.21.15.16:0/3298721789 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2aa006be60 0x7f2aa006e320 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4000f60 tx=0x7f2aa4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:21.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.583+0000 7f2ab0ff9640 1 -- 172.21.15.16:0/3298721789 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2ab415b270 con 0x7f2ab40d0530 2024-04-24T23:50:21.768 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.768+0000 7f2aba942640 1 -- 172.21.15.16:0/3298721789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2ab40d02c0 con 0x7f2ab40d0530 2024-04-24T23:50:21.769 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.770+0000 7f2ab0ff9640 1 -- 172.21.15.16:0/3298721789 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) ==== 74+0+16930 (secure 0 0 0) 0x7f2ab40d02c0 con 0x7f2ab40d0530 2024-04-24T23:50:21.769 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:21.770 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":39,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:50:20.068762+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:50:17.026573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-24T23:50:17.415963+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:50:21.771 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.771+0000 7f2aba942640 1 -- 172.21.15.16:0/3298721789 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2aa006be60 msgr2=0x7f2aa006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.771+0000 7f2aba942640 1 --2- 172.21.15.16:0/3298721789 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2aa006be60 0x7f2aa006e320 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4000f60 tx=0x7f2aa4009040 comp rx=0 tx=0).stop 2024-04-24T23:50:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.771+0000 7f2aba942640 1 -- 172.21.15.16:0/3298721789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab40d0530 msgr2=0x7f2ab414d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.771+0000 7f2aba942640 1 --2- 172.21.15.16:0/3298721789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab40d0530 0x7f2ab414d940 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8000c00 tx=0x7f2aa80046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.772+0000 7f2aba942640 1 -- 172.21.15.16:0/3298721789 shutdown_connections 2024-04-24T23:50:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.772+0000 7f2aba942640 1 --2- 172.21.15.16:0/3298721789 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2aa006be60 0x7f2aa006e320 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.772+0000 7f2aba942640 1 --2- 172.21.15.16:0/3298721789 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ab415df20 0x7f2ab415fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.772+0000 7f2aba942640 1 --2- 172.21.15.16:0/3298721789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ab40d0530 0x7f2ab414d940 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.772+0000 7f2aba942640 1 --2- 172.21.15.16:0/3298721789 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ab4072110 0x7f2ab414e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.772+0000 7f2aba942640 1 -- 172.21.15.16:0/3298721789 >> 172.21.15.16:0/3298721789 conn(0x7f2ab40d0ac0 msgr2=0x7f2ab4157670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.772+0000 7f2aba942640 1 -- 172.21.15.16:0/3298721789 shutdown_connections 2024-04-24T23:50:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.772+0000 7f2aba942640 1 -- 172.21.15.16:0/3298721789 wait complete. 2024-04-24T23:50:21.782 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-24T23:50:21.782 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:50:21.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.883+0000 7fe31f734640 1 Processor -- start 2024-04-24T23:50:21.883 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.884+0000 7fe31f734640 1 -- start start 2024-04-24T23:50:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.884+0000 7fe31f734640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe31815df20 0x7fe318071a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.885+0000 7fe31f734640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3180cfe50 0x7fe318071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.885+0000 7fe31f734640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3180b6a70 con 0x7fe3180d0530 2024-04-24T23:50:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.885+0000 7fe31f734640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fe3180cf460 con 0x7fe31815df20 2024-04-24T23:50:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.885+0000 7fe31f734640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe3180b6820 con 0x7fe3180cfe50 2024-04-24T23:50:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.885+0000 7fe31d4d0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe31815df20 0x7fe318071a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.885+0000 7fe31cccf640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe3180d0530 0x7fe3180ad8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51718/0 (socket says 172.21.15.16:51718) 2024-04-24T23:50:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.885+0000 7fe31cccf640 1 -- 172.21.15.16:0/2920204393 learned_addr learned my addr 172.21.15.16:0/2920204393 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.885+0000 7fe31dcd1640 1 --2- 172.21.15.16:0/2920204393 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3180cfe50 0x7fe318071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.885+0000 7fe31d4d0640 1 -- 172.21.15.16:0/2920204393 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3180cfe50 msgr2=0x7fe318071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.885+0000 7fe31d4d0640 1 --2- 172.21.15.16:0/2920204393 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3180cfe50 0x7fe318071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.885+0000 7fe31d4d0640 1 -- 172.21.15.16:0/2920204393 >> v1:172.21.15.16:6789/0 conn(0x7fe3180d0530 legacy=0x7fe3180ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.885 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.885+0000 7fe30ffff640 1 -- 172.21.15.16:0/2920204393 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 475663065 0 0) 0x7fe3180b6a70 con 0x7fe3180d0530 2024-04-24T23:50:21.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.885+0000 7fe31d4d0640 1 -- 172.21.15.16:0/2920204393 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3180ce6a0 con 0x7fe31815df20 2024-04-24T23:50:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.886+0000 7fe31d4d0640 1 --2- 172.21.15.16:0/2920204393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe31815df20 0x7fe318071a00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe3140097a0 tx=0x7fe31402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab3d52a2e10608f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.886+0000 7fe30ffff640 1 -- 172.21.15.16:0/2920204393 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314037070 con 0x7fe31815df20 2024-04-24T23:50:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.886+0000 7fe30ffff640 1 -- 172.21.15.16:0/2920204393 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe314032440 con 0x7fe31815df20 2024-04-24T23:50:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.886+0000 7fe30ffff640 1 -- 172.21.15.16:0/2920204393 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314032810 con 0x7fe31815df20 2024-04-24T23:50:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.886+0000 7fe31f734640 1 -- 172.21.15.16:0/2920204393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe31815df20 msgr2=0x7fe318071a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.886+0000 7fe31f734640 1 --2- 172.21.15.16:0/2920204393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe31815df20 0x7fe318071a00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fe3140097a0 tx=0x7fe31402a180 comp rx=0 tx=0).stop 2024-04-24T23:50:21.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.886+0000 7fe31f734640 1 -- 172.21.15.16:0/2920204393 shutdown_connections 2024-04-24T23:50:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.886+0000 7fe31f734640 1 --2- 172.21.15.16:0/2920204393 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3180cfe50 0x7fe318071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.886+0000 7fe31f734640 1 --2- 172.21.15.16:0/2920204393 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe31815df20 0x7fe318071a00 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.887+0000 7fe31f734640 1 -- 172.21.15.16:0/2920204393 >> 172.21.15.16:0/2920204393 conn(0x7fe3180d0ac0 msgr2=0x7fe3180d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.887+0000 7fe31f734640 1 -- 172.21.15.16:0/2920204393 shutdown_connections 2024-04-24T23:50:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.887+0000 7fe31f734640 1 -- 172.21.15.16:0/2920204393 wait complete. 2024-04-24T23:50:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.888+0000 7fe31f734640 1 Processor -- start 2024-04-24T23:50:21.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.888+0000 7fe31f734640 1 -- start start 2024-04-24T23:50:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.888+0000 7fe31f734640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3180cfe50 0x7fe31815d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.888+0000 7fe31f734640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3180d0530 0x7fe3181519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31f734640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe31815df20 0x7fe318160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31f734640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe3180cef40 con 0x7fe31815df20 2024-04-24T23:50:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31f734640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe3180cec60 con 0x7fe3180cfe50 2024-04-24T23:50:21.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31f734640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe3180b4030 con 0x7fe3180d0530 2024-04-24T23:50:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31dcd1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe31815df20 0x7fe318160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31dcd1640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe31815df20 0x7fe318160350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40350/0 (socket says 172.21.15.16:40350) 2024-04-24T23:50:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31dcd1640 1 -- 172.21.15.16:0/1534343790 learned_addr learned my addr 172.21.15.16:0/1534343790 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31d4d0640 1 --2- 172.21.15.16:0/1534343790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3180cfe50 0x7fe31815d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31cccf640 1 --2- 172.21.15.16:0/1534343790 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3180d0530 0x7fe3181519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31dcd1640 1 -- 172.21.15.16:0/1534343790 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3180d0530 msgr2=0x7fe3181519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31dcd1640 1 --2- 172.21.15.16:0/1534343790 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3180d0530 0x7fe3181519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31dcd1640 1 -- 172.21.15.16:0/1534343790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3180cfe50 msgr2=0x7fe31815d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31dcd1640 1 --2- 172.21.15.16:0/1534343790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3180cfe50 0x7fe31815d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31dcd1640 1 -- 172.21.15.16:0/1534343790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe318153150 con 0x7fe31815df20 2024-04-24T23:50:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31dcd1640 1 --2- 172.21.15.16:0/1534343790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe31815df20 0x7fe318160350 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe310007e70 tx=0x7fe31000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe30dffb640 1 -- 172.21.15.16:0/1534343790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe310014620 con 0x7fe31815df20 2024-04-24T23:50:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe30dffb640 1 -- 172.21.15.16:0/1534343790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe310019070 con 0x7fe31815df20 2024-04-24T23:50:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe31f734640 1 -- 172.21.15.16:0/1534343790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe318153430 con 0x7fe31815df20 2024-04-24T23:50:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.889+0000 7fe30dffb640 1 -- 172.21.15.16:0/1534343790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe310014a20 con 0x7fe31815df20 2024-04-24T23:50:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.890+0000 7fe31f734640 1 -- 172.21.15.16:0/1534343790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe31815e5e0 con 0x7fe31815df20 2024-04-24T23:50:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.890+0000 7fe31f734640 1 -- 172.21.15.16:0/1534343790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe318072f30 con 0x7fe31815df20 2024-04-24T23:50:21.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.891+0000 7fe30dffb640 1 -- 172.21.15.16:0/1534343790 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe310014bc0 con 0x7fe31815df20 2024-04-24T23:50:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.891+0000 7fe30dffb640 1 --2- 172.21.15.16:0/1534343790 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe2fc06be30 0x7fe2fc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.891+0000 7fe30dffb640 1 -- 172.21.15.16:0/1534343790 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fe31008b940 con 0x7fe31815df20 2024-04-24T23:50:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.891+0000 7fe31d4d0640 1 --2- 172.21.15.16:0/1534343790 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe2fc06be30 0x7fe2fc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:21.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.892+0000 7fe31d4d0640 1 --2- 172.21.15.16:0/1534343790 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe2fc06be30 0x7fe2fc06e2f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe314034000 tx=0x7fe31402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:21.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:21.896+0000 7fe30dffb640 1 -- 172.21.15.16:0/1534343790 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe318072f30 con 0x7fe31815df20 2024-04-24T23:50:22.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.074+0000 7fe31f734640 1 -- 172.21.15.16:0/1534343790 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe31806f8f0 con 0x7fe2fc06be30 2024-04-24T23:50:22.080 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.080+0000 7fe30dffb640 1 -- 172.21.15.16:0/1534343790 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240021 (secure 0 0 0) 0x7fe31806f8f0 con 0x7fe2fc06be30 2024-04-24T23:50:22.081 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:22.089 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-24T23:50:22.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.083+0000 7fe31f734640 1 -- 172.21.15.16:0/1534343790 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe2fc06be30 msgr2=0x7fe2fc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.092 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.083+0000 7fe31f734640 1 --2- 172.21.15.16:0/1534343790 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe2fc06be30 0x7fe2fc06e2f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe314034000 tx=0x7fe31402e000 comp rx=0 tx=0).stop 2024-04-24T23:50:22.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.083+0000 7fe31f734640 1 -- 172.21.15.16:0/1534343790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe31815df20 msgr2=0x7fe318160350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.083+0000 7fe31f734640 1 --2- 172.21.15.16:0/1534343790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe31815df20 0x7fe318160350 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe310007e70 tx=0x7fe31000e4b0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.083+0000 7fe31f734640 1 -- 172.21.15.16:0/1534343790 shutdown_connections 2024-04-24T23:50:22.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.083+0000 7fe31f734640 1 --2- 172.21.15.16:0/1534343790 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe2fc06be30 0x7fe2fc06e2f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.083+0000 7fe31f734640 1 --2- 172.21.15.16:0/1534343790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe31815df20 0x7fe318160350 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.083+0000 7fe31f734640 1 --2- 172.21.15.16:0/1534343790 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3180d0530 0x7fe3181519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.083+0000 7fe31f734640 1 --2- 172.21.15.16:0/1534343790 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3180cfe50 0x7fe31815d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.083+0000 7fe31f734640 1 -- 172.21.15.16:0/1534343790 >> 172.21.15.16:0/1534343790 conn(0x7fe3180d0ac0 msgr2=0x7fe3180ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:22.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.083+0000 7fe31f734640 1 -- 172.21.15.16:0/1534343790 shutdown_connections 2024-04-24T23:50:22.093 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.083+0000 7fe31f734640 1 -- 172.21.15.16:0/1534343790 wait complete. 2024-04-24T23:50:22.095 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-24T23:50:21.140012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253800,"kb_used_data":4308,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720920,"statfs":{"total":773094113280,"available":772834222080,"internally_reserved":0,"allocated":4411392,"data_stored":2530721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.077434"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067185+0000","last_change":"2024-04-24T23:50:17.067185+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067185+0000","last_undegraded":"2024-04-24T23:50:17.067185+0000","last_fullsized":"2024-04-24T23:50:17.067185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.066237+0000","last_change":"2024-04-24T23:50:17.066237+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.066237+0000","last_undegraded":"2024-04-24T23:50:17.066237+0000","last_fullsized":"2024-04-24T23:50:17.066237+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067422+0000","last_change":"2024-04-24T23:50:17.067422+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067422+0000","last_undegraded":"2024-04-24T23:50:17.067422+0000","last_fullsized":"2024-04-24T23:50:17.067422+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067661+0000","last_change":"2024-04-24T23:50:17.067661+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067661+0000","last_undegraded":"2024-04-24T23:50:17.067661+0000","last_fullsized":"2024-04-24T23:50:17.067661+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.068746+0000","last_change":"2024-04-24T23:50:17.068746+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.068746+0000","last_undegraded":"2024-04-24T23:50:17.068746+0000","last_fullsized":"2024-04-24T23:50:17.068746+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065001+0000","last_change":"2024-04-24T23:50:17.065001+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065001+0000","last_undegraded":"2024-04-24T23:50:17.065001+0000","last_fullsized":"2024-04-24T23:50:17.065001+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067343+0000","last_change":"2024-04-24T23:50:17.067343+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067343+0000","last_undegraded":"2024-04-24T23:50:17.067343+0000","last_fullsized":"2024-04-24T23:50:17.067343+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067258+0000","last_change":"2024-04-24T23:50:17.067258+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067258+0000","last_undegraded":"2024-04-24T23:50:17.067258+0000","last_fullsized":"2024-04-24T23:50:17.067258+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.063852+0000","last_change":"2024-04-24T23:50:17.063852+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.063852+0000","last_undegraded":"2024-04-24T23:50:17.063852+0000","last_fullsized":"2024-04-24T23:50:17.063852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.069350+0000","last_change":"2024-04-24T23:50:17.069350+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.069350+0000","last_undegraded":"2024-04-24T23:50:17.069350+0000","last_fullsized":"2024-04-24T23:50:17.069350+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.069401+0000","last_change":"2024-04-24T23:50:17.069401+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.069401+0000","last_undegraded":"2024-04-24T23:50:17.069401+0000","last_fullsized":"2024-04-24T23:50:17.069401+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.063840+0000","last_change":"2024-04-24T23:50:17.063840+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.063840+0000","last_undegraded":"2024-04-24T23:50:17.063840+0000","last_fullsized":"2024-04-24T23:50:17.063840+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.063783+0000","last_change":"2024-04-24T23:50:17.063783+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.063783+0000","last_undegraded":"2024-04-24T23:50:17.063783+0000","last_fullsized":"2024-04-24T23:50:17.063783+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067632+0000","last_change":"2024-04-24T23:50:17.067632+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067632+0000","last_undegraded":"2024-04-24T23:50:17.067632+0000","last_fullsized":"2024-04-24T23:50:17.067632+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067165+0000","last_change":"2024-04-24T23:50:17.067165+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067165+0000","last_undegraded":"2024-04-24T23:50:17.067165+0000","last_fullsized":"2024-04-24T23:50:17.067165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065011+0000","last_change":"2024-04-24T23:50:17.065011+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065011+0000","last_undegraded":"2024-04-24T23:50:17.065011+0000","last_fullsized":"2024-04-24T23:50:17.065011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.070185+0000","last_change":"2024-04-24T23:50:17.070185+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.070185+0000","last_undegraded":"2024-04-24T23:50:17.070185+0000","last_fullsized":"2024-04-24T23:50:17.070185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067669+0000","last_change":"2024-04-24T23:50:17.067669+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067669+0000","last_undegraded":"2024-04-24T23:50:17.067669+0000","last_fullsized":"2024-04-24T23:50:17.067669+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065960+0000","last_change":"2024-04-24T23:50:17.065960+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065960+0000","last_undegraded":"2024-04-24T23:50:17.065960+0000","last_fullsized":"2024-04-24T23:50:17.065960+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067488+0000","last_change":"2024-04-24T23:50:17.067488+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067488+0000","last_undegraded":"2024-04-24T23:50:17.067488+0000","last_fullsized":"2024-04-24T23:50:17.067488+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065650+0000","last_change":"2024-04-24T23:50:17.065650+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065650+0000","last_undegraded":"2024-04-24T23:50:17.065650+0000","last_fullsized":"2024-04-24T23:50:17.065650+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065500+0000","last_change":"2024-04-24T23:50:17.065500+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065500+0000","last_undegraded":"2024-04-24T23:50:17.065500+0000","last_fullsized":"2024-04-24T23:50:17.065500+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.064992+0000","last_change":"2024-04-24T23:50:17.064992+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.064992+0000","last_undegraded":"2024-04-24T23:50:17.064992+0000","last_fullsized":"2024-04-24T23:50:17.064992+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065348+0000","last_change":"2024-04-24T23:50:17.065348+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065348+0000","last_undegraded":"2024-04-24T23:50:17.065348+0000","last_fullsized":"2024-04-24T23:50:17.065348+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.069772+0000","last_change":"2024-04-24T23:50:17.069772+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.069772+0000","last_undegraded":"2024-04-24T23:50:17.069772+0000","last_fullsized":"2024-04-24T23:50:17.069772+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065343+0000","last_change":"2024-04-24T23:50:17.065343+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065343+0000","last_undegraded":"2024-04-24T23:50:17.065343+0000","last_fullsized":"2024-04-24T23:50:17.065343+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065606+0000","last_change":"2024-04-24T23:50:17.065606+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065606+0000","last_undegraded":"2024-04-24T23:50:17.065606+0000","last_fullsized":"2024-04-24T23:50:17.065606+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065929+0000","last_change":"2024-04-24T23:50:17.065929+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065929+0000","last_undegraded":"2024-04-24T23:50:17.065929+0000","last_fullsized":"2024-04-24T23:50:17.065929+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067172+0000","last_change":"2024-04-24T23:50:17.067172+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067172+0000","last_undegraded":"2024-04-24T23:50:17.067172+0000","last_fullsized":"2024-04-24T23:50:17.067172+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067928+0000","last_change":"2024-04-24T23:50:17.067928+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067928+0000","last_undegraded":"2024-04-24T23:50:17.067928+0000","last_fullsized":"2024-04-24T23:50:17.067928+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.063847+0000","last_change":"2024-04-24T23:50:17.063847+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.063847+0000","last_undegraded":"2024-04-24T23:50:17.063847+0000","last_fullsized":"2024-04-24T23:50:17.063847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065148+0000","last_change":"2024-04-24T23:50:17.065148+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065148+0000","last_undegraded":"2024-04-24T23:50:17.065148+0000","last_fullsized":"2024-04-24T23:50:17.065148+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":200,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-24T23:50:17.065527+0000","last_change":"2024-04-24T23:48:45.082459+0000","last_active":"2024-04-24T23:50:17.065527+0000","last_peered":"2024-04-24T23:50:17.065527+0000","last_clean":"2024-04-24T23:50:17.065527+0000","last_became_active":"2024-04-24T23:48:45.081495+0000","last_became_peered":"2024-04-24T23:48:45.081495+0000","last_unstale":"2024-04-24T23:50:17.065527+0000","last_undegraded":"2024-04-24T23:50:17.065527+0000","last_fullsized":"2024-04-24T23:50:17.065527+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_clean_scrub_stamp":"2024-04-24T23:48:43.440993+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:53:20.832511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":995328,"data_stored":758340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":413696,"data_stored":174181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32160,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339680,"statfs":{"total":96636764160,"available":96603832320,"internally_reserved":0,"allocated":995328,"data_stored":758340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:50:22.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:50:22.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.241+0000 7f819184d640 1 Processor -- start 2024-04-24T23:50:22.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.241+0000 7f819184d640 1 -- start start 2024-04-24T23:50:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.242+0000 7f819184d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f818c15e020 0x7f818c0cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.242+0000 7f819184d640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f818c0b2bb0 0x7f818c0cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.242+0000 7f819184d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f818c0cfcf0 0x7f818c161bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.242+0000 7f819184d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f818c0b3390 con 0x7f818c0cfcf0 2024-04-24T23:50:22.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.242+0000 7f819184d640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f818c06bfc0 con 0x7f818c0b2bb0 2024-04-24T23:50:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.242+0000 7f819184d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f818c06c2a0 con 0x7f818c15e020 2024-04-24T23:50:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.242+0000 7f818b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f818c0cfcf0 0x7f818c161bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.242+0000 7f818b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f818c0cfcf0 0x7f818c161bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40364/0 (socket says 172.21.15.16:40364) 2024-04-24T23:50:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.242+0000 7f818b7fe640 1 -- 172.21.15.16:0/1547595190 learned_addr learned my addr 172.21.15.16:0/1547595190 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.243+0000 7f818affd640 1 --2- 172.21.15.16:0/1547595190 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f818c15e020 0x7f818c0cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.243+0000 7f818a7fc640 1 --2- 172.21.15.16:0/1547595190 >> v2:172.21.15.156:3300/0 conn(0x7f818c0b2bb0 0x7f818c0cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.243+0000 7f818b7fe640 1 -- 172.21.15.16:0/1547595190 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f818c15e020 msgr2=0x7f818c0cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.243+0000 7f818b7fe640 1 --2- 172.21.15.16:0/1547595190 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f818c15e020 0x7f818c0cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.243+0000 7f818b7fe640 1 -- 172.21.15.16:0/1547595190 >> v2:172.21.15.156:3300/0 conn(0x7f818c0b2bb0 msgr2=0x7f818c0cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.243+0000 7f818b7fe640 1 --2- 172.21.15.16:0/1547595190 >> v2:172.21.15.156:3300/0 conn(0x7f818c0b2bb0 0x7f818c0cf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.243+0000 7f818b7fe640 1 -- 172.21.15.16:0/1547595190 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818c06c580 con 0x7f818c0cfcf0 2024-04-24T23:50:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.243+0000 7f818a7fc640 1 --2- 172.21.15.16:0/1547595190 >> v2:172.21.15.156:3300/0 conn(0x7f818c0b2bb0 0x7f818c0cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.243+0000 7f818affd640 1 --2- 172.21.15.16:0/1547595190 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f818c15e020 0x7f818c0cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:22.244 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.243+0000 7f818b7fe640 1 --2- 172.21.15.16:0/1547595190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818c0cfcf0 0x7f818c161bf0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f818c0b6a00 tx=0x7f818402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9cc8779debb66 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.244+0000 7f8189ffb640 1 -- 172.21.15.16:0/1547595190 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8184037070 con 0x7f818c0cfcf0 2024-04-24T23:50:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.244+0000 7f8189ffb640 1 -- 172.21.15.16:0/1547595190 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8184032440 con 0x7f818c0cfcf0 2024-04-24T23:50:22.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.244+0000 7f8189ffb640 1 -- 172.21.15.16:0/1547595190 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8184032810 con 0x7f818c0cfcf0 2024-04-24T23:50:22.247 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.244+0000 7f819184d640 1 -- 172.21.15.16:0/1547595190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818c0cfcf0 msgr2=0x7f818c161bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.244+0000 7f819184d640 1 --2- 172.21.15.16:0/1547595190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818c0cfcf0 0x7f818c161bf0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f818c0b6a00 tx=0x7f818402a140 comp rx=0 tx=0).stop 2024-04-24T23:50:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.244+0000 7f819184d640 1 -- 172.21.15.16:0/1547595190 shutdown_connections 2024-04-24T23:50:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.244+0000 7f819184d640 1 --2- 172.21.15.16:0/1547595190 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818c0cfcf0 0x7f818c161bf0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.244+0000 7f819184d640 1 --2- 172.21.15.16:0/1547595190 >> v2:172.21.15.156:3300/0 conn(0x7f818c0b2bb0 0x7f818c0cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.244+0000 7f819184d640 1 --2- 172.21.15.16:0/1547595190 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f818c15e020 0x7f818c0cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.244+0000 7f819184d640 1 -- 172.21.15.16:0/1547595190 >> 172.21.15.16:0/1547595190 conn(0x7f818c0d3e10 msgr2=0x7f818c0d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.244+0000 7f819184d640 1 -- 172.21.15.16:0/1547595190 shutdown_connections 2024-04-24T23:50:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.244+0000 7f819184d640 1 -- 172.21.15.16:0/1547595190 wait complete. 2024-04-24T23:50:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.245+0000 7f819184d640 1 Processor -- start 2024-04-24T23:50:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.246+0000 7f819184d640 1 -- start start 2024-04-24T23:50:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.246+0000 7f819184d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818c0b2bb0 0x7f818c159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.246+0000 7f819184d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f818c0cfcf0 0x7f818c0b85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.248 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.246+0000 7f819184d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f818c15e020 0x7f818c0b8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.246+0000 7f819184d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f818c15e400 con 0x7f818c0b2bb0 2024-04-24T23:50:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.246+0000 7f819184d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f818c0d1a80 con 0x7f818c0cfcf0 2024-04-24T23:50:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.246+0000 7f819184d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f818c162130 con 0x7f818c15e020 2024-04-24T23:50:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.246+0000 7f818affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818c0b2bb0 0x7f818c159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.246+0000 7f818affd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818c0b2bb0 0x7f818c159ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40380/0 (socket says 172.21.15.16:40380) 2024-04-24T23:50:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.246+0000 7f818affd640 1 -- 172.21.15.16:0/739946591 learned_addr learned my addr 172.21.15.16:0/739946591 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f818b7fe640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f818c15e020 0x7f818c0b8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f818a7fc640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f818c0cfcf0 0x7f818c0b85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f818affd640 1 -- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f818c15e020 msgr2=0x7f818c0b8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f818affd640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f818c15e020 0x7f818c0b8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f818affd640 1 -- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f818c0cfcf0 msgr2=0x7f818c0b85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f818affd640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f818c0cfcf0 0x7f818c0b85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f818affd640 1 -- 172.21.15.16:0/739946591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f818c158a40 con 0x7f818c0b2bb0 2024-04-24T23:50:22.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f818a7fc640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f818c0cfcf0 0x7f818c0b85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f818b7fe640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f818c15e020 0x7f818c0b8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f818affd640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818c0b2bb0 0x7f818c159ae0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f817c00ce00 tx=0x7f817c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f8173fff640 1 -- 172.21.15.16:0/739946591 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c00f890 con 0x7f818c0b2bb0 2024-04-24T23:50:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f8173fff640 1 -- 172.21.15.16:0/739946591 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f817c00fa30 con 0x7f818c0b2bb0 2024-04-24T23:50:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f819184d640 1 -- 172.21.15.16:0/739946591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f818c158d20 con 0x7f818c0b2bb0 2024-04-24T23:50:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f8173fff640 1 -- 172.21.15.16:0/739946591 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f817c00fe40 con 0x7f818c0b2bb0 2024-04-24T23:50:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.247+0000 7f819184d640 1 -- 172.21.15.16:0/739946591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f818c1592e0 con 0x7f818c0b2bb0 2024-04-24T23:50:22.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.248+0000 7f819184d640 1 -- 172.21.15.16:0/739946591 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f818c06dfe0 con 0x7f818c0b2bb0 2024-04-24T23:50:22.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.249+0000 7f8173fff640 1 -- 172.21.15.16:0/739946591 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f817c018400 con 0x7f818c0b2bb0 2024-04-24T23:50:22.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.249+0000 7f8173fff640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f817806be30 0x7f817806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.249+0000 7f8173fff640 1 -- 172.21.15.16:0/739946591 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f817c08beb0 con 0x7f818c0b2bb0 2024-04-24T23:50:22.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.249+0000 7f818a7fc640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f817806be30 0x7f817806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.250+0000 7f818a7fc640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f817806be30 0x7f817806e2f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f818c14b910 tx=0x7f8180008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:22.257 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.253+0000 7f8173fff640 1 -- 172.21.15.16:0/739946591 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f817c014030 con 0x7f818c0b2bb0 2024-04-24T23:50:22.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.433+0000 7f819184d640 1 -- 172.21.15.16:0/739946591 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f818c0b4a60 con 0x7f817806be30 2024-04-24T23:50:22.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.439+0000 7f8173fff640 1 -- 172.21.15.16:0/739946591 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240021 (secure 0 0 0) 0x7f818c0b4a60 con 0x7f817806be30 2024-04-24T23:50:22.441 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-24T23:50:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.441+0000 7f819184d640 1 -- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f817806be30 msgr2=0x7f817806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.441+0000 7f819184d640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f817806be30 0x7f817806e2f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f818c14b910 tx=0x7f8180008040 comp rx=0 tx=0).stop 2024-04-24T23:50:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.441+0000 7f819184d640 1 -- 172.21.15.16:0/739946591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818c0b2bb0 msgr2=0x7f818c159ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.441+0000 7f819184d640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818c0b2bb0 0x7f818c159ae0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f817c00ce00 tx=0x7f817c00a650 comp rx=0 tx=0).stop 2024-04-24T23:50:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.441+0000 7f819184d640 1 -- 172.21.15.16:0/739946591 shutdown_connections 2024-04-24T23:50:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.441+0000 7f819184d640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f817806be30 0x7f817806e2f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.441+0000 7f819184d640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f818c15e020 0x7f818c0b8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.441+0000 7f819184d640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f818c0cfcf0 0x7f818c0b85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.441+0000 7f819184d640 1 --2- 172.21.15.16:0/739946591 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f818c0b2bb0 0x7f818c159ae0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.441+0000 7f819184d640 1 -- 172.21.15.16:0/739946591 >> 172.21.15.16:0/739946591 conn(0x7f818c0d3e10 msgr2=0x7f818c0b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.442+0000 7f819184d640 1 -- 172.21.15.16:0/739946591 shutdown_connections 2024-04-24T23:50:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.442+0000 7f819184d640 1 -- 172.21.15.16:0/739946591 wait complete. 2024-04-24T23:50:22.452 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-24T23:50:21.140012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253800,"kb_used_data":4308,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720920,"statfs":{"total":773094113280,"available":772834222080,"internally_reserved":0,"allocated":4411392,"data_stored":2530721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.077434"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067185+0000","last_change":"2024-04-24T23:50:17.067185+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067185+0000","last_undegraded":"2024-04-24T23:50:17.067185+0000","last_fullsized":"2024-04-24T23:50:17.067185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.066237+0000","last_change":"2024-04-24T23:50:17.066237+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.066237+0000","last_undegraded":"2024-04-24T23:50:17.066237+0000","last_fullsized":"2024-04-24T23:50:17.066237+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067422+0000","last_change":"2024-04-24T23:50:17.067422+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067422+0000","last_undegraded":"2024-04-24T23:50:17.067422+0000","last_fullsized":"2024-04-24T23:50:17.067422+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067661+0000","last_change":"2024-04-24T23:50:17.067661+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067661+0000","last_undegraded":"2024-04-24T23:50:17.067661+0000","last_fullsized":"2024-04-24T23:50:17.067661+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.068746+0000","last_change":"2024-04-24T23:50:17.068746+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.068746+0000","last_undegraded":"2024-04-24T23:50:17.068746+0000","last_fullsized":"2024-04-24T23:50:17.068746+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065001+0000","last_change":"2024-04-24T23:50:17.065001+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065001+0000","last_undegraded":"2024-04-24T23:50:17.065001+0000","last_fullsized":"2024-04-24T23:50:17.065001+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067343+0000","last_change":"2024-04-24T23:50:17.067343+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067343+0000","last_undegraded":"2024-04-24T23:50:17.067343+0000","last_fullsized":"2024-04-24T23:50:17.067343+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067258+0000","last_change":"2024-04-24T23:50:17.067258+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067258+0000","last_undegraded":"2024-04-24T23:50:17.067258+0000","last_fullsized":"2024-04-24T23:50:17.067258+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.063852+0000","last_change":"2024-04-24T23:50:17.063852+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.063852+0000","last_undegraded":"2024-04-24T23:50:17.063852+0000","last_fullsized":"2024-04-24T23:50:17.063852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.069350+0000","last_change":"2024-04-24T23:50:17.069350+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.069350+0000","last_undegraded":"2024-04-24T23:50:17.069350+0000","last_fullsized":"2024-04-24T23:50:17.069350+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.069401+0000","last_change":"2024-04-24T23:50:17.069401+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.069401+0000","last_undegraded":"2024-04-24T23:50:17.069401+0000","last_fullsized":"2024-04-24T23:50:17.069401+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.063840+0000","last_change":"2024-04-24T23:50:17.063840+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.063840+0000","last_undegraded":"2024-04-24T23:50:17.063840+0000","last_fullsized":"2024-04-24T23:50:17.063840+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.063783+0000","last_change":"2024-04-24T23:50:17.063783+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.063783+0000","last_undegraded":"2024-04-24T23:50:17.063783+0000","last_fullsized":"2024-04-24T23:50:17.063783+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067632+0000","last_change":"2024-04-24T23:50:17.067632+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067632+0000","last_undegraded":"2024-04-24T23:50:17.067632+0000","last_fullsized":"2024-04-24T23:50:17.067632+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067165+0000","last_change":"2024-04-24T23:50:17.067165+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067165+0000","last_undegraded":"2024-04-24T23:50:17.067165+0000","last_fullsized":"2024-04-24T23:50:17.067165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065011+0000","last_change":"2024-04-24T23:50:17.065011+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065011+0000","last_undegraded":"2024-04-24T23:50:17.065011+0000","last_fullsized":"2024-04-24T23:50:17.065011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.070185+0000","last_change":"2024-04-24T23:50:17.070185+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.070185+0000","last_undegraded":"2024-04-24T23:50:17.070185+0000","last_fullsized":"2024-04-24T23:50:17.070185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067669+0000","last_change":"2024-04-24T23:50:17.067669+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067669+0000","last_undegraded":"2024-04-24T23:50:17.067669+0000","last_fullsized":"2024-04-24T23:50:17.067669+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065960+0000","last_change":"2024-04-24T23:50:17.065960+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065960+0000","last_undegraded":"2024-04-24T23:50:17.065960+0000","last_fullsized":"2024-04-24T23:50:17.065960+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067488+0000","last_change":"2024-04-24T23:50:17.067488+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067488+0000","last_undegraded":"2024-04-24T23:50:17.067488+0000","last_fullsized":"2024-04-24T23:50:17.067488+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065650+0000","last_change":"2024-04-24T23:50:17.065650+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065650+0000","last_undegraded":"2024-04-24T23:50:17.065650+0000","last_fullsized":"2024-04-24T23:50:17.065650+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065500+0000","last_change":"2024-04-24T23:50:17.065500+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065500+0000","last_undegraded":"2024-04-24T23:50:17.065500+0000","last_fullsized":"2024-04-24T23:50:17.065500+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.064992+0000","last_change":"2024-04-24T23:50:17.064992+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.064992+0000","last_undegraded":"2024-04-24T23:50:17.064992+0000","last_fullsized":"2024-04-24T23:50:17.064992+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065348+0000","last_change":"2024-04-24T23:50:17.065348+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065348+0000","last_undegraded":"2024-04-24T23:50:17.065348+0000","last_fullsized":"2024-04-24T23:50:17.065348+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.069772+0000","last_change":"2024-04-24T23:50:17.069772+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.069772+0000","last_undegraded":"2024-04-24T23:50:17.069772+0000","last_fullsized":"2024-04-24T23:50:17.069772+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065343+0000","last_change":"2024-04-24T23:50:17.065343+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065343+0000","last_undegraded":"2024-04-24T23:50:17.065343+0000","last_fullsized":"2024-04-24T23:50:17.065343+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065606+0000","last_change":"2024-04-24T23:50:17.065606+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065606+0000","last_undegraded":"2024-04-24T23:50:17.065606+0000","last_fullsized":"2024-04-24T23:50:17.065606+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065929+0000","last_change":"2024-04-24T23:50:17.065929+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065929+0000","last_undegraded":"2024-04-24T23:50:17.065929+0000","last_fullsized":"2024-04-24T23:50:17.065929+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067172+0000","last_change":"2024-04-24T23:50:17.067172+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067172+0000","last_undegraded":"2024-04-24T23:50:17.067172+0000","last_fullsized":"2024-04-24T23:50:17.067172+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067928+0000","last_change":"2024-04-24T23:50:17.067928+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067928+0000","last_undegraded":"2024-04-24T23:50:17.067928+0000","last_fullsized":"2024-04-24T23:50:17.067928+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.063847+0000","last_change":"2024-04-24T23:50:17.063847+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.063847+0000","last_undegraded":"2024-04-24T23:50:17.063847+0000","last_fullsized":"2024-04-24T23:50:17.063847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065148+0000","last_change":"2024-04-24T23:50:17.065148+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065148+0000","last_undegraded":"2024-04-24T23:50:17.065148+0000","last_fullsized":"2024-04-24T23:50:17.065148+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":200,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-24T23:50:17.065527+0000","last_change":"2024-04-24T23:48:45.082459+0000","last_active":"2024-04-24T23:50:17.065527+0000","last_peered":"2024-04-24T23:50:17.065527+0000","last_clean":"2024-04-24T23:50:17.065527+0000","last_became_active":"2024-04-24T23:48:45.081495+0000","last_became_peered":"2024-04-24T23:48:45.081495+0000","last_unstale":"2024-04-24T23:50:17.065527+0000","last_undegraded":"2024-04-24T23:50:17.065527+0000","last_fullsized":"2024-04-24T23:50:17.065527+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_clean_scrub_stamp":"2024-04-24T23:48:43.440993+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:53:20.832511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":995328,"data_stored":758340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":413696,"data_stored":174181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32160,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339680,"statfs":{"total":96636764160,"available":96603832320,"internally_reserved":0,"allocated":995328,"data_stored":758340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:50:22.459 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:50:22.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.594+0000 7f04d53f4640 1 Processor -- start 2024-04-24T23:50:22.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.594+0000 7f04d53f4640 1 -- start start 2024-04-24T23:50:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.594+0000 7f04d53f4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04d01457b0 0x7f04d00ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.594+0000 7f04d53f4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04d015fe90 con 0x7f04d015f990 2024-04-24T23:50:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.594+0000 7f04d53f4640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04d0160170 con 0x7f04d0144b20 2024-04-24T23:50:22.595 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.594+0000 7f04d53f4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f04d0160450 con 0x7f04d01457b0 2024-04-24T23:50:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.595+0000 7f04ce7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04d01457b0 0x7f04d00ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.595+0000 7f04ce7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04d01457b0 0x7f04d00ce680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40396/0 (socket says 172.21.15.16:40396) 2024-04-24T23:50:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.595+0000 7f04ce7fc640 1 -- 172.21.15.16:0/1900834699 learned_addr learned my addr 172.21.15.16:0/1900834699 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.595+0000 7f04cf7fe640 1 --1- 172.21.15.16:0/1900834699 >> v1:172.21.15.16:6789/0 conn(0x7f04d015f990 0x7f04d006da40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51726/0 (socket says 172.21.15.16:51726) 2024-04-24T23:50:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.595+0000 7f04ce7fc640 1 -- 172.21.15.16:0/1900834699 >> v1:172.21.15.156:6789/0 conn(0x7f04d0144b20 legacy=0x7f04d0144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.595+0000 7f04ce7fc640 1 -- 172.21.15.16:0/1900834699 >> v1:172.21.15.16:6789/0 conn(0x7f04d015f990 legacy=0x7f04d006da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.595+0000 7f04ce7fc640 1 -- 172.21.15.16:0/1900834699 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d0160730 con 0x7f04d01457b0 2024-04-24T23:50:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.595+0000 7f04cdffb640 1 -- 172.21.15.16:0/1900834699 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3573344856 0 0) 0x7f04d015fe90 con 0x7f04d015f990 2024-04-24T23:50:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.596+0000 7f04ce7fc640 1 --2- 172.21.15.16:0/1900834699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d01457b0 0x7f04d00ce680 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009870 tx=0x7f04c402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e52e76530613b31 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.596+0000 7f04cdffb640 1 -- 172.21.15.16:0/1900834699 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c4037070 con 0x7f04d01457b0 2024-04-24T23:50:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.596+0000 7f04cdffb640 1 -- 172.21.15.16:0/1900834699 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04c4032440 con 0x7f04d01457b0 2024-04-24T23:50:22.596 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.596+0000 7f04cdffb640 1 -- 172.21.15.16:0/1900834699 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c4032810 con 0x7f04d01457b0 2024-04-24T23:50:22.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.596+0000 7f04d53f4640 1 -- 172.21.15.16:0/1900834699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d01457b0 msgr2=0x7f04d00ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.596+0000 7f04d53f4640 1 --2- 172.21.15.16:0/1900834699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d01457b0 0x7f04d00ce680 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009870 tx=0x7f04c402a210 comp rx=0 tx=0).stop 2024-04-24T23:50:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.596+0000 7f04d53f4640 1 -- 172.21.15.16:0/1900834699 shutdown_connections 2024-04-24T23:50:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.596+0000 7f04d53f4640 1 --2- 172.21.15.16:0/1900834699 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d01457b0 0x7f04d00ce680 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.596+0000 7f04d53f4640 1 -- 172.21.15.16:0/1900834699 >> 172.21.15.16:0/1900834699 conn(0x7f04d00b2000 msgr2=0x7f04d00cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.596+0000 7f04d53f4640 1 -- 172.21.15.16:0/1900834699 shutdown_connections 2024-04-24T23:50:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.596+0000 7f04d53f4640 1 -- 172.21.15.16:0/1900834699 wait complete. 2024-04-24T23:50:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.598+0000 7f04d53f4640 1 Processor -- start 2024-04-24T23:50:22.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.598+0000 7f04d53f4640 1 -- start start 2024-04-24T23:50:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.598+0000 7f04d53f4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04d0144b20 0x7f04d00cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.598+0000 7f04d53f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d01457b0 0x7f04d00cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.598+0000 7f04d53f4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04d015f990 0x7f04d013fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.598+0000 7f04d53f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f04d00cebc0 con 0x7f04d01457b0 2024-04-24T23:50:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.598+0000 7f04d53f4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f04d00b3390 con 0x7f04d0144b20 2024-04-24T23:50:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.598+0000 7f04d53f4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f04d00ced40 con 0x7f04d015f990 2024-04-24T23:50:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04ce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d01457b0 0x7f04d00cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04ce7fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d01457b0 0x7f04d00cb250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40408/0 (socket says 172.21.15.16:40408) 2024-04-24T23:50:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04ce7fc640 1 -- 172.21.15.16:0/1907302094 learned_addr learned my addr 172.21.15.16:0/1907302094 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04ceffd640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04d0144b20 0x7f04d00cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04cf7fe640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04d015f990 0x7f04d013fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04ce7fc640 1 -- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04d015f990 msgr2=0x7f04d013fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04ce7fc640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04d015f990 0x7f04d013fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04ce7fc640 1 -- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04d0144b20 msgr2=0x7f04d00cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04ce7fc640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04d0144b20 0x7f04d00cdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04ce7fc640 1 -- 172.21.15.16:0/1907302094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d00c9620 con 0x7f04d01457b0 2024-04-24T23:50:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04ceffd640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04d0144b20 0x7f04d00cdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:22.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04ce7fc640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d01457b0 0x7f04d00cb250 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009870 tx=0x7f04c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04b77fe640 1 -- 172.21.15.16:0/1907302094 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c4041070 con 0x7f04d01457b0 2024-04-24T23:50:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04b77fe640 1 -- 172.21.15.16:0/1907302094 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04c403c870 con 0x7f04d01457b0 2024-04-24T23:50:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04d53f4640 1 -- 172.21.15.16:0/1907302094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d00c9340 con 0x7f04d01457b0 2024-04-24T23:50:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04b77fe640 1 -- 172.21.15.16:0/1907302094 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c4037040 con 0x7f04d01457b0 2024-04-24T23:50:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.599+0000 7f04d53f4640 1 -- 172.21.15.16:0/1907302094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d013c6c0 con 0x7f04d01457b0 2024-04-24T23:50:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.600+0000 7f04d53f4640 1 -- 172.21.15.16:0/1907302094 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04d00731f0 con 0x7f04d01457b0 2024-04-24T23:50:22.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.601+0000 7f04b77fe640 1 -- 172.21.15.16:0/1907302094 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f04c4044070 con 0x7f04d01457b0 2024-04-24T23:50:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.601+0000 7f04b77fe640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f04bc06be30 0x7f04bc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.601+0000 7f04b77fe640 1 -- 172.21.15.16:0/1907302094 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f04c40abc70 con 0x7f04d01457b0 2024-04-24T23:50:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.602+0000 7f04ceffd640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f04bc06be30 0x7f04bc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.602+0000 7f04ceffd640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f04bc06be30 0x7f04bc06e2f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f04c0003eb0 tx=0x7f04c0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:22.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.606+0000 7f04b77fe640 1 -- 172.21.15.16:0/1907302094 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f04d00c9620 con 0x7f04d01457b0 2024-04-24T23:50:22.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.785+0000 7f04d53f4640 1 -- 172.21.15.16:0/1907302094 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f04d004f5e0 con 0x7f04bc06be30 2024-04-24T23:50:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.791+0000 7f04b77fe640 1 -- 172.21.15.16:0/1907302094 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+240021 (secure 0 0 0) 0x7f04c00539b0 con 0x7f04bc06be30 2024-04-24T23:50:22.793 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:22.794 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-24T23:50:22.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.794+0000 7f04d53f4640 1 -- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f04bc06be30 msgr2=0x7f04bc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.794+0000 7f04d53f4640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f04bc06be30 0x7f04bc06e2f0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f04c0003eb0 tx=0x7f04c0009040 comp rx=0 tx=0).stop 2024-04-24T23:50:22.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.794+0000 7f04d53f4640 1 -- 172.21.15.16:0/1907302094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d01457b0 msgr2=0x7f04d00cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.794+0000 7f04d53f4640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d01457b0 0x7f04d00cb250 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f04c4009870 tx=0x7f04c40046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.794+0000 7f04d53f4640 1 -- 172.21.15.16:0/1907302094 shutdown_connections 2024-04-24T23:50:22.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.794+0000 7f04d53f4640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f04bc06be30 0x7f04bc06e2f0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.794+0000 7f04d53f4640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f04d015f990 0x7f04d013fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.794+0000 7f04d53f4640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d01457b0 0x7f04d00cb250 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.794+0000 7f04d53f4640 1 --2- 172.21.15.16:0/1907302094 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04d0144b20 0x7f04d00cdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.794+0000 7f04d53f4640 1 -- 172.21.15.16:0/1907302094 >> 172.21.15.16:0/1907302094 conn(0x7f04d00b2000 msgr2=0x7f04d00716e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:22.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.794+0000 7f04d53f4640 1 -- 172.21.15.16:0/1907302094 shutdown_connections 2024-04-24T23:50:22.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.794+0000 7f04d53f4640 1 -- 172.21.15.16:0/1907302094 wait complete. 2024-04-24T23:50:22.805 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-04-24T23:50:21.140012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":253800,"kb_used_data":4308,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720920,"statfs":{"total":773094113280,"available":772834222080,"internally_reserved":0,"allocated":4411392,"data_stored":2530721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.077434"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067185+0000","last_change":"2024-04-24T23:50:17.067185+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067185+0000","last_undegraded":"2024-04-24T23:50:17.067185+0000","last_fullsized":"2024-04-24T23:50:17.067185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.066237+0000","last_change":"2024-04-24T23:50:17.066237+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.066237+0000","last_undegraded":"2024-04-24T23:50:17.066237+0000","last_fullsized":"2024-04-24T23:50:17.066237+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067422+0000","last_change":"2024-04-24T23:50:17.067422+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067422+0000","last_undegraded":"2024-04-24T23:50:17.067422+0000","last_fullsized":"2024-04-24T23:50:17.067422+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067661+0000","last_change":"2024-04-24T23:50:17.067661+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067661+0000","last_undegraded":"2024-04-24T23:50:17.067661+0000","last_fullsized":"2024-04-24T23:50:17.067661+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.068746+0000","last_change":"2024-04-24T23:50:17.068746+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.068746+0000","last_undegraded":"2024-04-24T23:50:17.068746+0000","last_fullsized":"2024-04-24T23:50:17.068746+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065001+0000","last_change":"2024-04-24T23:50:17.065001+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065001+0000","last_undegraded":"2024-04-24T23:50:17.065001+0000","last_fullsized":"2024-04-24T23:50:17.065001+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067343+0000","last_change":"2024-04-24T23:50:17.067343+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067343+0000","last_undegraded":"2024-04-24T23:50:17.067343+0000","last_fullsized":"2024-04-24T23:50:17.067343+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067258+0000","last_change":"2024-04-24T23:50:17.067258+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067258+0000","last_undegraded":"2024-04-24T23:50:17.067258+0000","last_fullsized":"2024-04-24T23:50:17.067258+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.063852+0000","last_change":"2024-04-24T23:50:17.063852+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.063852+0000","last_undegraded":"2024-04-24T23:50:17.063852+0000","last_fullsized":"2024-04-24T23:50:17.063852+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.069350+0000","last_change":"2024-04-24T23:50:17.069350+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.069350+0000","last_undegraded":"2024-04-24T23:50:17.069350+0000","last_fullsized":"2024-04-24T23:50:17.069350+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.069401+0000","last_change":"2024-04-24T23:50:17.069401+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.069401+0000","last_undegraded":"2024-04-24T23:50:17.069401+0000","last_fullsized":"2024-04-24T23:50:17.069401+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.063840+0000","last_change":"2024-04-24T23:50:17.063840+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.063840+0000","last_undegraded":"2024-04-24T23:50:17.063840+0000","last_fullsized":"2024-04-24T23:50:17.063840+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.063783+0000","last_change":"2024-04-24T23:50:17.063783+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.063783+0000","last_undegraded":"2024-04-24T23:50:17.063783+0000","last_fullsized":"2024-04-24T23:50:17.063783+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067632+0000","last_change":"2024-04-24T23:50:17.067632+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067632+0000","last_undegraded":"2024-04-24T23:50:17.067632+0000","last_fullsized":"2024-04-24T23:50:17.067632+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067165+0000","last_change":"2024-04-24T23:50:17.067165+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067165+0000","last_undegraded":"2024-04-24T23:50:17.067165+0000","last_fullsized":"2024-04-24T23:50:17.067165+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065011+0000","last_change":"2024-04-24T23:50:17.065011+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065011+0000","last_undegraded":"2024-04-24T23:50:17.065011+0000","last_fullsized":"2024-04-24T23:50:17.065011+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.070185+0000","last_change":"2024-04-24T23:50:17.070185+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.070185+0000","last_undegraded":"2024-04-24T23:50:17.070185+0000","last_fullsized":"2024-04-24T23:50:17.070185+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067669+0000","last_change":"2024-04-24T23:50:17.067669+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067669+0000","last_undegraded":"2024-04-24T23:50:17.067669+0000","last_fullsized":"2024-04-24T23:50:17.067669+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065960+0000","last_change":"2024-04-24T23:50:17.065960+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065960+0000","last_undegraded":"2024-04-24T23:50:17.065960+0000","last_fullsized":"2024-04-24T23:50:17.065960+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067488+0000","last_change":"2024-04-24T23:50:17.067488+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067488+0000","last_undegraded":"2024-04-24T23:50:17.067488+0000","last_fullsized":"2024-04-24T23:50:17.067488+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065650+0000","last_change":"2024-04-24T23:50:17.065650+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065650+0000","last_undegraded":"2024-04-24T23:50:17.065650+0000","last_fullsized":"2024-04-24T23:50:17.065650+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065500+0000","last_change":"2024-04-24T23:50:17.065500+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065500+0000","last_undegraded":"2024-04-24T23:50:17.065500+0000","last_fullsized":"2024-04-24T23:50:17.065500+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.064992+0000","last_change":"2024-04-24T23:50:17.064992+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.064992+0000","last_undegraded":"2024-04-24T23:50:17.064992+0000","last_fullsized":"2024-04-24T23:50:17.064992+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065348+0000","last_change":"2024-04-24T23:50:17.065348+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065348+0000","last_undegraded":"2024-04-24T23:50:17.065348+0000","last_fullsized":"2024-04-24T23:50:17.065348+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.069772+0000","last_change":"2024-04-24T23:50:17.069772+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.069772+0000","last_undegraded":"2024-04-24T23:50:17.069772+0000","last_fullsized":"2024-04-24T23:50:17.069772+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065343+0000","last_change":"2024-04-24T23:50:17.065343+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065343+0000","last_undegraded":"2024-04-24T23:50:17.065343+0000","last_fullsized":"2024-04-24T23:50:17.065343+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065606+0000","last_change":"2024-04-24T23:50:17.065606+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065606+0000","last_undegraded":"2024-04-24T23:50:17.065606+0000","last_fullsized":"2024-04-24T23:50:17.065606+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065929+0000","last_change":"2024-04-24T23:50:17.065929+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065929+0000","last_undegraded":"2024-04-24T23:50:17.065929+0000","last_fullsized":"2024-04-24T23:50:17.065929+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067172+0000","last_change":"2024-04-24T23:50:17.067172+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067172+0000","last_undegraded":"2024-04-24T23:50:17.067172+0000","last_fullsized":"2024-04-24T23:50:17.067172+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.067928+0000","last_change":"2024-04-24T23:50:17.067928+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.067928+0000","last_undegraded":"2024-04-24T23:50:17.067928+0000","last_fullsized":"2024-04-24T23:50:17.067928+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.063847+0000","last_change":"2024-04-24T23:50:17.063847+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.063847+0000","last_undegraded":"2024-04-24T23:50:17.063847+0000","last_fullsized":"2024-04-24T23:50:17.063847+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-04-24T23:50:18.062426+0000","last_change":"2024-04-24T23:50:18.062426+0000","last_active":"2024-04-24T23:50:18.062426+0000","last_peered":"2024-04-24T23:50:18.062426+0000","last_clean":"2024-04-24T23:50:18.062426+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:18.062426+0000","last_undegraded":"2024-04-24T23:50:18.062426+0000","last_fullsized":"2024-04-24T23:50:18.062426+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":36,"state":"creating+peering","last_fresh":"2024-04-24T23:50:17.065148+0000","last_change":"2024-04-24T23:50:17.065148+0000","last_active":"2024-04-24T23:50:17.060744+0000","last_peered":"2024-04-24T23:50:17.060744+0000","last_clean":"2024-04-24T23:50:17.060744+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-04-24T23:50:17.065148+0000","last_undegraded":"2024-04-24T23:50:17.065148+0000","last_fullsized":"2024-04-24T23:50:17.065148+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":200,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-04-24T23:50:17.065527+0000","last_change":"2024-04-24T23:48:45.082459+0000","last_active":"2024-04-24T23:50:17.065527+0000","last_peered":"2024-04-24T23:50:17.065527+0000","last_clean":"2024-04-24T23:50:17.065527+0000","last_became_active":"2024-04-24T23:48:45.081495+0000","last_became_peered":"2024-04-24T23:48:45.081495+0000","last_unstale":"2024-04-24T23:50:17.065527+0000","last_undegraded":"2024-04-24T23:50:17.065527+0000","last_fullsized":"2024-04-24T23:50:17.065527+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_clean_scrub_stamp":"2024-04-24T23:48:43.440993+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:53:20.832511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32172,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339668,"statfs":{"total":96636764160,"available":96603820032,"internally_reserved":0,"allocated":995328,"data_stored":758340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":404,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":413696,"data_stored":174181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31572,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340268,"statfs":{"total":96636764160,"available":96604434432,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31584,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340256,"statfs":{"total":96636764160,"available":96604422144,"internally_reserved":0,"allocated":401408,"data_stored":167972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32160,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339680,"statfs":{"total":96636764160,"available":96603832320,"internally_reserved":0,"allocated":995328,"data_stored":758340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:50:22.810 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3f is not active+clean 2024-04-24T23:50:22.810 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.810 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3e is not active+clean 2024-04-24T23:50:22.810 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.810 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3d is not active+clean 2024-04-24T23:50:22.810 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.810 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3c is not active+clean 2024-04-24T23:50:22.810 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.811 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3b is not active+clean 2024-04-24T23:50:22.811 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.811 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3a is not active+clean 2024-04-24T23:50:22.811 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.812 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.39 is not active+clean 2024-04-24T23:50:22.812 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.812 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.38 is not active+clean 2024-04-24T23:50:22.812 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.812 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.37 is not active+clean 2024-04-24T23:50:22.812 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.812 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.36 is not active+clean 2024-04-24T23:50:22.812 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.812 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.35 is not active+clean 2024-04-24T23:50:22.812 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.812 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.34 is not active+clean 2024-04-24T23:50:22.813 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.813 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.33 is not active+clean 2024-04-24T23:50:22.813 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.813 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.32 is not active+clean 2024-04-24T23:50:22.813 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.813 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.31 is not active+clean 2024-04-24T23:50:22.813 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.813 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.30 is not active+clean 2024-04-24T23:50:22.813 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.813 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2f is not active+clean 2024-04-24T23:50:22.813 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.814 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2e is not active+clean 2024-04-24T23:50:22.814 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.814 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2d is not active+clean 2024-04-24T23:50:22.814 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.814 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2c is not active+clean 2024-04-24T23:50:22.814 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.814 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2b is not active+clean 2024-04-24T23:50:22.814 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.814 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2a is not active+clean 2024-04-24T23:50:22.814 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.814 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.29 is not active+clean 2024-04-24T23:50:22.815 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.815 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.28 is not active+clean 2024-04-24T23:50:22.815 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.815 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.27 is not active+clean 2024-04-24T23:50:22.815 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.815 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.26 is not active+clean 2024-04-24T23:50:22.815 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.815 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.25 is not active+clean 2024-04-24T23:50:22.815 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.816 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.24 is not active+clean 2024-04-24T23:50:22.816 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.816 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.23 is not active+clean 2024-04-24T23:50:22.816 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.816 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.22 is not active+clean 2024-04-24T23:50:22.816 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.816 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.21 is not active+clean 2024-04-24T23:50:22.816 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.816 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.20 is not active+clean 2024-04-24T23:50:22.816 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.817 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1c is not active+clean 2024-04-24T23:50:22.817 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.817 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2024-04-24T23:50:22.817 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.067185+0000', 'last_change': '2024-04-24T23:50:17.067185+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.067185+0000', 'last_undegraded': '2024-04-24T23:50:17.067185+0000', 'last_fullsized': '2024-04-24T23:50:17.067185+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-24T23:50:22.817 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1d is not active+clean 2024-04-24T23:50:22.817 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.817 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2024-04-24T23:50:22.817 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.066237+0000', 'last_change': '2024-04-24T23:50:17.066237+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.066237+0000', 'last_undegraded': '2024-04-24T23:50:17.066237+0000', 'last_fullsized': '2024-04-24T23:50:17.066237+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-24T23:50:22.818 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1e is not active+clean 2024-04-24T23:50:22.818 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.818 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2024-04-24T23:50:22.818 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.067422+0000', 'last_change': '2024-04-24T23:50:17.067422+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.067422+0000', 'last_undegraded': '2024-04-24T23:50:17.067422+0000', 'last_fullsized': '2024-04-24T23:50:17.067422+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-24T23:50:22.818 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1f is not active+clean 2024-04-24T23:50:22.819 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.819 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2024-04-24T23:50:22.819 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.067661+0000', 'last_change': '2024-04-24T23:50:17.067661+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.067661+0000', 'last_undegraded': '2024-04-24T23:50:17.067661+0000', 'last_fullsized': '2024-04-24T23:50:17.067661+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-24T23:50:22.819 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.18 is not active+clean 2024-04-24T23:50:22.819 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.819 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2024-04-24T23:50:22.819 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.068746+0000', 'last_change': '2024-04-24T23:50:17.068746+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.068746+0000', 'last_undegraded': '2024-04-24T23:50:17.068746+0000', 'last_fullsized': '2024-04-24T23:50:17.068746+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T23:50:22.819 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.19 is not active+clean 2024-04-24T23:50:22.819 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.820 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2024-04-24T23:50:22.820 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.065001+0000', 'last_change': '2024-04-24T23:50:17.065001+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.065001+0000', 'last_undegraded': '2024-04-24T23:50:17.065001+0000', 'last_fullsized': '2024-04-24T23:50:17.065001+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T23:50:22.820 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1a is not active+clean 2024-04-24T23:50:22.820 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.820 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2024-04-24T23:50:22.820 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.067343+0000', 'last_change': '2024-04-24T23:50:17.067343+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.067343+0000', 'last_undegraded': '2024-04-24T23:50:17.067343+0000', 'last_fullsized': '2024-04-24T23:50:17.067343+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-24T23:50:22.820 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1b is not active+clean 2024-04-24T23:50:22.820 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.821 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2024-04-24T23:50:22.821 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.067258+0000', 'last_change': '2024-04-24T23:50:17.067258+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.067258+0000', 'last_undegraded': '2024-04-24T23:50:17.067258+0000', 'last_fullsized': '2024-04-24T23:50:17.067258+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-24T23:50:22.821 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.14 is not active+clean 2024-04-24T23:50:22.821 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.822 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2024-04-24T23:50:22.822 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.063852+0000', 'last_change': '2024-04-24T23:50:17.063852+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.063852+0000', 'last_undegraded': '2024-04-24T23:50:17.063852+0000', 'last_fullsized': '2024-04-24T23:50:17.063852+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-24T23:50:22.822 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.15 is not active+clean 2024-04-24T23:50:22.822 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.822 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2024-04-24T23:50:22.822 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.069350+0000', 'last_change': '2024-04-24T23:50:17.069350+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.069350+0000', 'last_undegraded': '2024-04-24T23:50:17.069350+0000', 'last_fullsized': '2024-04-24T23:50:17.069350+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T23:50:22.822 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.16 is not active+clean 2024-04-24T23:50:22.822 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2024-04-24T23:50:22.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.069401+0000', 'last_change': '2024-04-24T23:50:17.069401+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.069401+0000', 'last_undegraded': '2024-04-24T23:50:17.069401+0000', 'last_fullsized': '2024-04-24T23:50:17.069401+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T23:50:22.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.17 is not active+clean 2024-04-24T23:50:22.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2024-04-24T23:50:22.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.063840+0000', 'last_change': '2024-04-24T23:50:17.063840+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.063840+0000', 'last_undegraded': '2024-04-24T23:50:17.063840+0000', 'last_fullsized': '2024-04-24T23:50:17.063840+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-24T23:50:22.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.10 is not active+clean 2024-04-24T23:50:22.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.823 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2024-04-24T23:50:22.823 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.063783+0000', 'last_change': '2024-04-24T23:50:17.063783+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.063783+0000', 'last_undegraded': '2024-04-24T23:50:17.063783+0000', 'last_fullsized': '2024-04-24T23:50:17.063783+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-24T23:50:22.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.11 is not active+clean 2024-04-24T23:50:22.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2024-04-24T23:50:22.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.067632+0000', 'last_change': '2024-04-24T23:50:17.067632+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.067632+0000', 'last_undegraded': '2024-04-24T23:50:17.067632+0000', 'last_fullsized': '2024-04-24T23:50:17.067632+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-24T23:50:22.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.12 is not active+clean 2024-04-24T23:50:22.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2024-04-24T23:50:22.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.067165+0000', 'last_change': '2024-04-24T23:50:17.067165+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.067165+0000', 'last_undegraded': '2024-04-24T23:50:17.067165+0000', 'last_fullsized': '2024-04-24T23:50:17.067165+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-24T23:50:22.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.13 is not active+clean 2024-04-24T23:50:22.824 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.824 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2024-04-24T23:50:22.825 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.065011+0000', 'last_change': '2024-04-24T23:50:17.065011+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.065011+0000', 'last_undegraded': '2024-04-24T23:50:17.065011+0000', 'last_fullsized': '2024-04-24T23:50:17.065011+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T23:50:22.825 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.c is not active+clean 2024-04-24T23:50:22.825 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.825 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2024-04-24T23:50:22.825 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.070185+0000', 'last_change': '2024-04-24T23:50:17.070185+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.070185+0000', 'last_undegraded': '2024-04-24T23:50:17.070185+0000', 'last_fullsized': '2024-04-24T23:50:17.070185+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T23:50:22.825 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.d is not active+clean 2024-04-24T23:50:22.825 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.825 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2024-04-24T23:50:22.825 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.067669+0000', 'last_change': '2024-04-24T23:50:17.067669+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.067669+0000', 'last_undegraded': '2024-04-24T23:50:17.067669+0000', 'last_fullsized': '2024-04-24T23:50:17.067669+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-24T23:50:22.825 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.e is not active+clean 2024-04-24T23:50:22.825 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.826 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2024-04-24T23:50:22.826 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.065960+0000', 'last_change': '2024-04-24T23:50:17.065960+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.065960+0000', 'last_undegraded': '2024-04-24T23:50:17.065960+0000', 'last_fullsized': '2024-04-24T23:50:17.065960+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-24T23:50:22.826 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.f is not active+clean 2024-04-24T23:50:22.826 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.826 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2024-04-24T23:50:22.826 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.067488+0000', 'last_change': '2024-04-24T23:50:17.067488+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.067488+0000', 'last_undegraded': '2024-04-24T23:50:17.067488+0000', 'last_fullsized': '2024-04-24T23:50:17.067488+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-04-24T23:50:22.826 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.8 is not active+clean 2024-04-24T23:50:22.826 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.827 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2024-04-24T23:50:22.827 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.065650+0000', 'last_change': '2024-04-24T23:50:17.065650+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.065650+0000', 'last_undegraded': '2024-04-24T23:50:17.065650+0000', 'last_fullsized': '2024-04-24T23:50:17.065650+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-24T23:50:22.827 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.9 is not active+clean 2024-04-24T23:50:22.827 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.827 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2024-04-24T23:50:22.827 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.065500+0000', 'last_change': '2024-04-24T23:50:17.065500+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.065500+0000', 'last_undegraded': '2024-04-24T23:50:17.065500+0000', 'last_fullsized': '2024-04-24T23:50:17.065500+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-24T23:50:22.827 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.a is not active+clean 2024-04-24T23:50:22.827 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.827 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2024-04-24T23:50:22.827 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.064992+0000', 'last_change': '2024-04-24T23:50:17.064992+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.064992+0000', 'last_undegraded': '2024-04-24T23:50:17.064992+0000', 'last_fullsized': '2024-04-24T23:50:17.064992+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T23:50:22.827 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.b is not active+clean 2024-04-24T23:50:22.828 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.828 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2024-04-24T23:50:22.828 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.065348+0000', 'last_change': '2024-04-24T23:50:17.065348+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.065348+0000', 'last_undegraded': '2024-04-24T23:50:17.065348+0000', 'last_fullsized': '2024-04-24T23:50:17.065348+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-24T23:50:22.828 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.4 is not active+clean 2024-04-24T23:50:22.828 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.828 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2024-04-24T23:50:22.828 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.069772+0000', 'last_change': '2024-04-24T23:50:17.069772+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.069772+0000', 'last_undegraded': '2024-04-24T23:50:17.069772+0000', 'last_fullsized': '2024-04-24T23:50:17.069772+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-04-24T23:50:22.828 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.5 is not active+clean 2024-04-24T23:50:22.828 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.829 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2024-04-24T23:50:22.829 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.065343+0000', 'last_change': '2024-04-24T23:50:17.065343+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.065343+0000', 'last_undegraded': '2024-04-24T23:50:17.065343+0000', 'last_fullsized': '2024-04-24T23:50:17.065343+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-04-24T23:50:22.829 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.6 is not active+clean 2024-04-24T23:50:22.829 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.829 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2024-04-24T23:50:22.829 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.065606+0000', 'last_change': '2024-04-24T23:50:17.065606+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.065606+0000', 'last_undegraded': '2024-04-24T23:50:17.065606+0000', 'last_fullsized': '2024-04-24T23:50:17.065606+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-24T23:50:22.829 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.0 is not active+clean 2024-04-24T23:50:22.829 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.829 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2024-04-24T23:50:22.829 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.065929+0000', 'last_change': '2024-04-24T23:50:17.065929+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.065929+0000', 'last_undegraded': '2024-04-24T23:50:17.065929+0000', 'last_fullsized': '2024-04-24T23:50:17.065929+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-04-24T23:50:22.829 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.1 is not active+clean 2024-04-24T23:50:22.830 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.830 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2024-04-24T23:50:22.830 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.067172+0000', 'last_change': '2024-04-24T23:50:17.067172+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.067172+0000', 'last_undegraded': '2024-04-24T23:50:17.067172+0000', 'last_fullsized': '2024-04-24T23:50:17.067172+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-04-24T23:50:22.830 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.2 is not active+clean 2024-04-24T23:50:22.830 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.830 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2024-04-24T23:50:22.830 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.067928+0000', 'last_change': '2024-04-24T23:50:17.067928+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.067928+0000', 'last_undegraded': '2024-04-24T23:50:17.067928+0000', 'last_fullsized': '2024-04-24T23:50:17.067928+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-04-24T23:50:22.830 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.3 is not active+clean 2024-04-24T23:50:22.830 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.830 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2024-04-24T23:50:22.830 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.063847+0000', 'last_change': '2024-04-24T23:50:17.063847+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.063847+0000', 'last_undegraded': '2024-04-24T23:50:17.063847+0000', 'last_fullsized': '2024-04-24T23:50:17.063847+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-04-24T23:50:22.831 INFO:tasks.cephfs.filesystem.ceph_manager:PG 6.7 is not active+clean 2024-04-24T23:50:22.831 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '6.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-04-24T23:50:18.062426+0000', 'last_change': '2024-04-24T23:50:18.062426+0000', 'last_active': '2024-04-24T23:50:18.062426+0000', 'last_peered': '2024-04-24T23:50:18.062426+0000', 'last_clean': '2024-04-24T23:50:18.062426+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:18.062426+0000', 'last_undegraded': '2024-04-24T23:50:18.062426+0000', 'last_fullsized': '2024-04-24T23:50:18.062426+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:18.062426+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-04-24T23:50:22.831 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2024-04-24T23:50:22.831 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 36, 'state': 'creating+peering', 'last_fresh': '2024-04-24T23:50:17.065148+0000', 'last_change': '2024-04-24T23:50:17.065148+0000', 'last_active': '2024-04-24T23:50:17.060744+0000', 'last_peered': '2024-04-24T23:50:17.060744+0000', 'last_clean': '2024-04-24T23:50:17.060744+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-04-24T23:50:17.065148+0000', 'last_undegraded': '2024-04-24T23:50:17.065148+0000', 'last_fullsized': '2024-04-24T23:50:17.065148+0000', 'mapping_epoch': 36, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 36, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'last_clean_scrub_stamp': '2024-04-24T23:50:17.060744+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-04-24T23:50:22.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-24T23:50:22.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.947+0000 7ffa6254f640 1 Processor -- start 2024-04-24T23:50:22.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.947+0000 7ffa6254f640 1 -- start start 2024-04-24T23:50:22.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.947+0000 7ffa6254f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa5c15df20 0x7ffa5c071a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.947+0000 7ffa6254f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ffa5c0d0530 0x7ffa5c071f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.948 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.947+0000 7ffa6254f640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa5c0b6a70 con 0x7ffa5c0cfe50 2024-04-24T23:50:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.947+0000 7ffa6254f640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ffa5c0cf460 con 0x7ffa5c0d0530 2024-04-24T23:50:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.947+0000 7ffa6254f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ffa5c0b6820 con 0x7ffa5c15df20 2024-04-24T23:50:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.948+0000 7ffa5bfff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa5c15df20 0x7ffa5c071a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.948+0000 7ffa5bfff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa5c15df20 0x7ffa5c071a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:59730/0 (socket says 172.21.15.16:59730) 2024-04-24T23:50:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.948+0000 7ffa5bfff640 1 -- 172.21.15.16:0/386201543 learned_addr learned my addr 172.21.15.16:0/386201543 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.948+0000 7ffa5b7fe640 1 --2- 172.21.15.16:0/386201543 >> v2:172.21.15.156:3300/0 conn(0x7ffa5c0d0530 0x7ffa5c071f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.948+0000 7ffa5bfff640 1 -- 172.21.15.16:0/386201543 >> v2:172.21.15.156:3300/0 conn(0x7ffa5c0d0530 msgr2=0x7ffa5c071f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.948+0000 7ffa5bfff640 1 --2- 172.21.15.16:0/386201543 >> v2:172.21.15.156:3300/0 conn(0x7ffa5c0d0530 0x7ffa5c071f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.948+0000 7ffa5bfff640 1 -- 172.21.15.16:0/386201543 >> v1:172.21.15.156:6789/0 conn(0x7ffa5c0cfe50 legacy=0x7ffa5c0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.948+0000 7ffa5bfff640 1 -- 172.21.15.16:0/386201543 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa5c0ce6a0 con 0x7ffa5c15df20 2024-04-24T23:50:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.949+0000 7ffa5bfff640 1 --2- 172.21.15.16:0/386201543 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa5c15df20 0x7ffa5c071a00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c009870 tx=0x7ffa4c02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d659efe1fbca149 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.949+0000 7ffa5affd640 1 -- 172.21.15.16:0/386201543 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c037070 con 0x7ffa5c15df20 2024-04-24T23:50:22.949 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.949+0000 7ffa5affd640 1 -- 172.21.15.16:0/386201543 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa4c032440 con 0x7ffa5c15df20 2024-04-24T23:50:22.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.949+0000 7ffa5affd640 1 -- 172.21.15.16:0/386201543 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4c032810 con 0x7ffa5c15df20 2024-04-24T23:50:22.955 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.949+0000 7ffa6254f640 1 -- 172.21.15.16:0/386201543 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa5c15df20 msgr2=0x7ffa5c071a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.949+0000 7ffa6254f640 1 --2- 172.21.15.16:0/386201543 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa5c15df20 0x7ffa5c071a00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c009870 tx=0x7ffa4c02a270 comp rx=0 tx=0).stop 2024-04-24T23:50:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.950+0000 7ffa6254f640 1 -- 172.21.15.16:0/386201543 shutdown_connections 2024-04-24T23:50:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.950+0000 7ffa6254f640 1 --2- 172.21.15.16:0/386201543 >> v2:172.21.15.156:3300/0 conn(0x7ffa5c0d0530 0x7ffa5c071f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.950+0000 7ffa6254f640 1 --2- 172.21.15.16:0/386201543 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa5c15df20 0x7ffa5c071a00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.950+0000 7ffa6254f640 1 -- 172.21.15.16:0/386201543 >> 172.21.15.16:0/386201543 conn(0x7ffa5c0d0ac0 msgr2=0x7ffa5c0d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.950+0000 7ffa6254f640 1 -- 172.21.15.16:0/386201543 shutdown_connections 2024-04-24T23:50:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.950+0000 7ffa6254f640 1 -- 172.21.15.16:0/386201543 wait complete. 2024-04-24T23:50:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.951+0000 7ffa6254f640 1 Processor -- start 2024-04-24T23:50:22.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.951+0000 7ffa6254f640 1 -- start start 2024-04-24T23:50:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa6254f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa5c0cfe50 0x7ffa5c156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa6254f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa5c0d0530 0x7ffa5c157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa6254f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa5c15df20 0x7ffa5c1553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa6254f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5c0cef40 con 0x7ffa5c15df20 2024-04-24T23:50:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa6254f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5c0cec60 con 0x7ffa5c0cfe50 2024-04-24T23:50:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa6254f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ffa5c0b4030 con 0x7ffa5c0d0530 2024-04-24T23:50:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa60aec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa5c15df20 0x7ffa5c1553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa60aec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa5c15df20 0x7ffa5c1553b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40410/0 (socket says 172.21.15.16:40410) 2024-04-24T23:50:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa60aec640 1 -- 172.21.15.16:0/3334413690 learned_addr learned my addr 172.21.15.16:0/3334413690 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa5bfff640 1 --2- 172.21.15.16:0/3334413690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa5c0cfe50 0x7ffa5c156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa5b7fe640 1 --2- 172.21.15.16:0/3334413690 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa5c0d0530 0x7ffa5c157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa60aec640 1 -- 172.21.15.16:0/3334413690 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa5c0d0530 msgr2=0x7ffa5c157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa60aec640 1 --2- 172.21.15.16:0/3334413690 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa5c0d0530 0x7ffa5c157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa60aec640 1 -- 172.21.15.16:0/3334413690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa5c0cfe50 msgr2=0x7ffa5c156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa60aec640 1 --2- 172.21.15.16:0/3334413690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa5c0cfe50 0x7ffa5c156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa60aec640 1 -- 172.21.15.16:0/3334413690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa5c157f10 con 0x7ffa5c15df20 2024-04-24T23:50:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa60aec640 1 --2- 172.21.15.16:0/3334413690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa5c15df20 0x7ffa5c1553b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400c7a0 tx=0x7ffa5400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa58ff9640 1 -- 172.21.15.16:0/3334413690 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa54003b70 con 0x7ffa5c15df20 2024-04-24T23:50:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.952+0000 7ffa58ff9640 1 -- 172.21.15.16:0/3334413690 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa54003d30 con 0x7ffa5c15df20 2024-04-24T23:50:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.953+0000 7ffa6254f640 1 -- 172.21.15.16:0/3334413690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa5c1581f0 con 0x7ffa5c15df20 2024-04-24T23:50:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.953+0000 7ffa58ff9640 1 -- 172.21.15.16:0/3334413690 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5400aa90 con 0x7ffa5c15df20 2024-04-24T23:50:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.953+0000 7ffa6254f640 1 -- 172.21.15.16:0/3334413690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa5c15e7f0 con 0x7ffa5c15df20 2024-04-24T23:50:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.953+0000 7ffa6254f640 1 -- 172.21.15.16:0/3334413690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa5c15e9f0 con 0x7ffa5c15df20 2024-04-24T23:50:22.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.954+0000 7ffa58ff9640 1 -- 172.21.15.16:0/3334413690 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ffa5400ac30 con 0x7ffa5c15df20 2024-04-24T23:50:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.954+0000 7ffa58ff9640 1 --2- 172.21.15.16:0/3334413690 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffa4806be60 0x7ffa4806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.955+0000 7ffa58ff9640 1 -- 172.21.15.16:0/3334413690 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7ffa5408af30 con 0x7ffa5c15df20 2024-04-24T23:50:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.955+0000 7ffa5bfff640 1 --2- 172.21.15.16:0/3334413690 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffa4806be60 0x7ffa4806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.955+0000 7ffa5bfff640 1 --2- 172.21.15.16:0/3334413690 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffa4806be60 0x7ffa4806e320 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c034000 tx=0x7ffa4c02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:22.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:22.959+0000 7ffa58ff9640 1 -- 172.21.15.16:0/3334413690 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ffa54059740 con 0x7ffa5c15df20 2024-04-24T23:50:23.051 INFO:tasks.ceph.mds.a.smithi016.stderr:2024-04-24T23:50:23.052+0000 7f439e3a5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T23:50:23.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.210+0000 7ffa6254f640 1 -- 172.21.15.16:0/3334413690 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7ffa5c06f8f0 con 0x7ffa5c15df20 2024-04-24T23:50:23.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.211+0000 7ffa58ff9640 1 -- 172.21.15.16:0/3334413690 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1109 (secure 0 0 0) 0x7ffa54059040 con 0x7ffa5c15df20 2024-04-24T23:50:23.211 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:23.211 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":112,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":39,"num_osds":8,"num_up_osds":8,"osd_up_since":1714002521,"num_in_osds":8,"osd_in_since":1714002513,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":259891200,"bytes_avail":772834222080,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":15,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5490}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-04-24T23:50:01.134276+0000","services":{}},"progress_events":{}} 2024-04-24T23:50:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.213+0000 7ffa6254f640 1 -- 172.21.15.16:0/3334413690 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffa4806be60 msgr2=0x7ffa4806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.213+0000 7ffa6254f640 1 --2- 172.21.15.16:0/3334413690 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffa4806be60 0x7ffa4806e320 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c034000 tx=0x7ffa4c02e000 comp rx=0 tx=0).stop 2024-04-24T23:50:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.213+0000 7ffa6254f640 1 -- 172.21.15.16:0/3334413690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa5c15df20 msgr2=0x7ffa5c1553b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.213+0000 7ffa6254f640 1 --2- 172.21.15.16:0/3334413690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa5c15df20 0x7ffa5c1553b0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ffa5400c7a0 tx=0x7ffa5400cc60 comp rx=0 tx=0).stop 2024-04-24T23:50:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.213+0000 7ffa6254f640 1 -- 172.21.15.16:0/3334413690 shutdown_connections 2024-04-24T23:50:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.213+0000 7ffa6254f640 1 --2- 172.21.15.16:0/3334413690 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ffa4806be60 0x7ffa4806e320 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.213+0000 7ffa6254f640 1 --2- 172.21.15.16:0/3334413690 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa5c15df20 0x7ffa5c1553b0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:23.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.213+0000 7ffa6254f640 1 --2- 172.21.15.16:0/3334413690 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ffa5c0d0530 0x7ffa5c157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:23.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.213+0000 7ffa6254f640 1 --2- 172.21.15.16:0/3334413690 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ffa5c0cfe50 0x7ffa5c156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:23.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.213+0000 7ffa6254f640 1 -- 172.21.15.16:0/3334413690 >> 172.21.15.16:0/3334413690 conn(0x7ffa5c0d0ac0 msgr2=0x7ffa5c0b3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:23.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.213+0000 7ffa6254f640 1 -- 172.21.15.16:0/3334413690 shutdown_connections 2024-04-24T23:50:23.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.213+0000 7ffa6254f640 1 -- 172.21.15.16:0/3334413690 wait complete. 2024-04-24T23:50:23.224 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-24T23:50:23.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:50:23.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.324+0000 7f74daeb3640 1 Processor -- start 2024-04-24T23:50:23.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.324+0000 7f74daeb3640 1 -- start start 2024-04-24T23:50:23.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.325+0000 7f74daeb3640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f74d40ce7c0 0x7f74d40cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:23.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.325+0000 7f74daeb3640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74d40cf3a0 con 0x7f74d4144740 2024-04-24T23:50:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.325+0000 7f74daeb3640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74d40cc8d0 con 0x7f74d4071a00 2024-04-24T23:50:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.325+0000 7f74daeb3640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f74d415c1f0 con 0x7f74d40ce7c0 2024-04-24T23:50:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.325+0000 7f74d9450640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f74d40ce7c0 0x7f74d40cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.325+0000 7f74d3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f74d4144740 0x7f74d415c3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51730/0 (socket says 172.21.15.16:51730) 2024-04-24T23:50:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.325+0000 7f74d9450640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f74d40ce7c0 0x7f74d40cec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40426/0 (socket says 172.21.15.16:40426) 2024-04-24T23:50:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.325+0000 7f74d3fff640 1 -- 172.21.15.16:0/798403533 learned_addr learned my addr 172.21.15.16:0/798403533 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.326+0000 7f74d9450640 1 -- 172.21.15.16:0/798403533 >> v1:172.21.15.156:6789/0 conn(0x7f74d4071a00 legacy=0x7f74d4071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.326+0000 7f74d9450640 1 -- 172.21.15.16:0/798403533 >> v1:172.21.15.16:6789/0 conn(0x7f74d4144740 legacy=0x7f74d415c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.326+0000 7f74d9450640 1 -- 172.21.15.16:0/798403533 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d40ccbb0 con 0x7f74d40ce7c0 2024-04-24T23:50:23.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.326+0000 7f74d37fe640 1 -- 172.21.15.16:0/798403533 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 783760676 0 0) 0x7f74d40cf3a0 con 0x7f74d4144740 2024-04-24T23:50:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.326+0000 7f74d9450640 1 --2- 172.21.15.16:0/798403533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74d40ce7c0 0x7f74d40cec10 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f74c40097a0 tx=0x7f74c402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc382f8dc896895e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.327+0000 7f74d37fe640 1 -- 172.21.15.16:0/798403533 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c4037070 con 0x7f74d40ce7c0 2024-04-24T23:50:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.327+0000 7f74d37fe640 1 -- 172.21.15.16:0/798403533 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74c4032440 con 0x7f74d40ce7c0 2024-04-24T23:50:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.327+0000 7f74d37fe640 1 -- 172.21.15.16:0/798403533 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74c4032810 con 0x7f74d40ce7c0 2024-04-24T23:50:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.327+0000 7f74daeb3640 1 -- 172.21.15.16:0/798403533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74d40ce7c0 msgr2=0x7f74d40cec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.327+0000 7f74daeb3640 1 --2- 172.21.15.16:0/798403533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74d40ce7c0 0x7f74d40cec10 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f74c40097a0 tx=0x7f74c402a230 comp rx=0 tx=0).stop 2024-04-24T23:50:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.327+0000 7f74daeb3640 1 -- 172.21.15.16:0/798403533 shutdown_connections 2024-04-24T23:50:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.327+0000 7f74daeb3640 1 --2- 172.21.15.16:0/798403533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74d40ce7c0 0x7f74d40cec10 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.327+0000 7f74daeb3640 1 -- 172.21.15.16:0/798403533 >> 172.21.15.16:0/798403533 conn(0x7f74d40b2000 msgr2=0x7f74d40cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.327+0000 7f74daeb3640 1 -- 172.21.15.16:0/798403533 shutdown_connections 2024-04-24T23:50:23.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.327+0000 7f74daeb3640 1 -- 172.21.15.16:0/798403533 wait complete. 2024-04-24T23:50:23.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.328+0000 7f74daeb3640 1 Processor -- start 2024-04-24T23:50:23.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.328+0000 7f74daeb3640 1 -- start start 2024-04-24T23:50:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.329+0000 7f74daeb3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74d4071a00 0x7f74d41f4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.329+0000 7f74daeb3640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f74d40ce7c0 0x7f74d415a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.329+0000 7f74daeb3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74d406d880 0x7f74d406dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.329+0000 7f74daeb3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f74d40cd170 con 0x7f74d4071a00 2024-04-24T23:50:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.329+0000 7f74daeb3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f74d40cf150 con 0x7f74d406d880 2024-04-24T23:50:23.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.329+0000 7f74daeb3640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f74d415cab0 con 0x7f74d40ce7c0 2024-04-24T23:50:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.329+0000 7f74d8c4f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74d4071a00 0x7f74d41f4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.329+0000 7f74d8c4f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74d4071a00 0x7f74d41f4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:40442/0 (socket says 172.21.15.16:40442) 2024-04-24T23:50:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.329+0000 7f74d8c4f640 1 -- 172.21.15.16:0/2085740611 learned_addr learned my addr 172.21.15.16:0/2085740611 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74d3fff640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f74d40ce7c0 0x7f74d415a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74d9450640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74d406d880 0x7f74d406dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74d8c4f640 1 -- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f74d40ce7c0 msgr2=0x7f74d415a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74d8c4f640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f74d40ce7c0 0x7f74d415a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74d8c4f640 1 -- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74d406d880 msgr2=0x7f74d406dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74d8c4f640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74d406d880 0x7f74d406dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74d8c4f640 1 -- 172.21.15.16:0/2085740611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74d41f5350 con 0x7f74d4071a00 2024-04-24T23:50:23.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74d3fff640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f74d40ce7c0 0x7f74d415a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74d9450640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74d406d880 0x7f74d406dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74d8c4f640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74d4071a00 0x7f74d41f4e10 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00c8a0 tx=0x7f74cc00cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74d17fa640 1 -- 172.21.15.16:0/2085740611 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc003b70 con 0x7f74d4071a00 2024-04-24T23:50:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74d17fa640 1 -- 172.21.15.16:0/2085740611 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74cc003d30 con 0x7f74d4071a00 2024-04-24T23:50:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74daeb3640 1 -- 172.21.15.16:0/2085740611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74d415aab0 con 0x7f74d4071a00 2024-04-24T23:50:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74d17fa640 1 -- 172.21.15.16:0/2085740611 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74cc00aa90 con 0x7f74d4071a00 2024-04-24T23:50:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.330+0000 7f74daeb3640 1 -- 172.21.15.16:0/2085740611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74d41392f0 con 0x7f74d4071a00 2024-04-24T23:50:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.331+0000 7f74daeb3640 1 -- 172.21.15.16:0/2085740611 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74d40731f0 con 0x7f74d4071a00 2024-04-24T23:50:23.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.331+0000 7f74d17fa640 1 -- 172.21.15.16:0/2085740611 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f74cc00ac30 con 0x7f74d4071a00 2024-04-24T23:50:23.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.332+0000 7f74d17fa640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f74c006be30 0x7f74c006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:23.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.332+0000 7f74d17fa640 1 -- 172.21.15.16:0/2085740611 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f74cc08af70 con 0x7f74d4071a00 2024-04-24T23:50:23.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.332+0000 7f74d3fff640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f74c006be30 0x7f74c006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:23.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.333+0000 7f74d3fff640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f74c006be30 0x7f74c006e2f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f74d41f5ea0 tx=0x7f74c8009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:23.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.336+0000 7f74d17fa640 1 -- 172.21.15.16:0/2085740611 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f74d415aab0 con 0x7f74d4071a00 2024-04-24T23:50:23.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.514+0000 7f74daeb3640 1 -- 172.21.15.16:0/2085740611 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f74d4138dc0 con 0x7f74c006be30 2024-04-24T23:50:23.520 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.521+0000 7f74d17fa640 1 -- 172.21.15.16:0/2085740611 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251955 (secure 0 0 0) 0x7f74d4138dc0 con 0x7f74c006be30 2024-04-24T23:50:23.524 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-24T23:50:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.524+0000 7f74daeb3640 1 -- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f74c006be30 msgr2=0x7f74c006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.524+0000 7f74daeb3640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f74c006be30 0x7f74c006e2f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f74d41f5ea0 tx=0x7f74c8009040 comp rx=0 tx=0).stop 2024-04-24T23:50:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.524+0000 7f74daeb3640 1 -- 172.21.15.16:0/2085740611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74d4071a00 msgr2=0x7f74d41f4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.524+0000 7f74daeb3640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74d4071a00 0x7f74d41f4e10 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f74cc00c8a0 tx=0x7f74cc00cd60 comp rx=0 tx=0).stop 2024-04-24T23:50:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.524+0000 7f74daeb3640 1 -- 172.21.15.16:0/2085740611 shutdown_connections 2024-04-24T23:50:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.524+0000 7f74daeb3640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f74c006be30 0x7f74c006e2f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.524+0000 7f74daeb3640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74d406d880 0x7f74d406dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.524+0000 7f74daeb3640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f74d40ce7c0 0x7f74d415a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.524+0000 7f74daeb3640 1 --2- 172.21.15.16:0/2085740611 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f74d4071a00 0x7f74d41f4e10 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.524+0000 7f74daeb3640 1 -- 172.21.15.16:0/2085740611 >> 172.21.15.16:0/2085740611 conn(0x7f74d40b2000 msgr2=0x7f74d415c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.524+0000 7f74daeb3640 1 -- 172.21.15.16:0/2085740611 shutdown_connections 2024-04-24T23:50:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:23.524+0000 7f74daeb3640 1 -- 172.21.15.16:0/2085740611 wait complete. 2024-04-24T23:50:23.534 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-04-24T23:50:23.140541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":110,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254184,"kb_used_data":4744,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720536,"statfs":{"total":773094113280,"available":772833828864,"internally_reserved":0,"allocated":4857856,"data_stored":2743212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001320"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788401+0000","last_change":"2024-04-24T23:50:19.096331+0000","last_active":"2024-04-24T23:50:20.788401+0000","last_peered":"2024-04-24T23:50:20.788401+0000","last_clean":"2024-04-24T23:50:20.788401+0000","last_became_active":"2024-04-24T23:50:19.095755+0000","last_became_peered":"2024-04-24T23:50:19.095755+0000","last_unstale":"2024-04-24T23:50:20.788401+0000","last_undegraded":"2024-04-24T23:50:20.788401+0000","last_fullsized":"2024-04-24T23:50:20.788401+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:16:23.638767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788156+0000","last_change":"2024-04-24T23:50:19.086054+0000","last_active":"2024-04-24T23:50:20.788156+0000","last_peered":"2024-04-24T23:50:20.788156+0000","last_clean":"2024-04-24T23:50:20.788156+0000","last_became_active":"2024-04-24T23:50:19.085407+0000","last_became_peered":"2024-04-24T23:50:19.085407+0000","last_unstale":"2024-04-24T23:50:20.788156+0000","last_undegraded":"2024-04-24T23:50:20.788156+0000","last_fullsized":"2024-04-24T23:50:20.788156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:44.423218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788242+0000","last_change":"2024-04-24T23:50:19.085231+0000","last_active":"2024-04-24T23:50:20.788242+0000","last_peered":"2024-04-24T23:50:20.788242+0000","last_clean":"2024-04-24T23:50:20.788242+0000","last_became_active":"2024-04-24T23:50:19.084968+0000","last_became_peered":"2024-04-24T23:50:19.084968+0000","last_unstale":"2024-04-24T23:50:20.788242+0000","last_undegraded":"2024-04-24T23:50:20.788242+0000","last_fullsized":"2024-04-24T23:50:20.788242+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:39:01.832638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788692+0000","last_change":"2024-04-24T23:50:19.090227+0000","last_active":"2024-04-24T23:50:20.788692+0000","last_peered":"2024-04-24T23:50:20.788692+0000","last_clean":"2024-04-24T23:50:20.788692+0000","last_became_active":"2024-04-24T23:50:19.089634+0000","last_became_peered":"2024-04-24T23:50:19.089634+0000","last_unstale":"2024-04-24T23:50:20.788692+0000","last_undegraded":"2024-04-24T23:50:20.788692+0000","last_fullsized":"2024-04-24T23:50:20.788692+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:02.733361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788678+0000","last_change":"2024-04-24T23:50:19.089138+0000","last_active":"2024-04-24T23:50:20.788678+0000","last_peered":"2024-04-24T23:50:20.788678+0000","last_clean":"2024-04-24T23:50:20.788678+0000","last_became_active":"2024-04-24T23:50:19.088868+0000","last_became_peered":"2024-04-24T23:50:19.088868+0000","last_unstale":"2024-04-24T23:50:20.788678+0000","last_undegraded":"2024-04-24T23:50:20.788678+0000","last_fullsized":"2024-04-24T23:50:20.788678+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:08:42.255450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.071816+0000","last_change":"2024-04-24T23:50:19.085098+0000","last_active":"2024-04-24T23:50:20.071816+0000","last_peered":"2024-04-24T23:50:20.071816+0000","last_clean":"2024-04-24T23:50:20.071816+0000","last_became_active":"2024-04-24T23:50:19.084806+0000","last_became_peered":"2024-04-24T23:50:19.084806+0000","last_unstale":"2024-04-24T23:50:20.071816+0000","last_undegraded":"2024-04-24T23:50:20.071816+0000","last_fullsized":"2024-04-24T23:50:20.071816+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:59.514575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.787948+0000","last_change":"2024-04-24T23:50:19.081837+0000","last_active":"2024-04-24T23:50:20.787948+0000","last_peered":"2024-04-24T23:50:20.787948+0000","last_clean":"2024-04-24T23:50:20.787948+0000","last_became_active":"2024-04-24T23:50:19.081526+0000","last_became_peered":"2024-04-24T23:50:19.081526+0000","last_unstale":"2024-04-24T23:50:20.787948+0000","last_undegraded":"2024-04-24T23:50:20.787948+0000","last_fullsized":"2024-04-24T23:50:20.787948+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:49:15.545291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.787850+0000","last_change":"2024-04-24T23:50:19.085713+0000","last_active":"2024-04-24T23:50:20.787850+0000","last_peered":"2024-04-24T23:50:20.787850+0000","last_clean":"2024-04-24T23:50:20.787850+0000","last_became_active":"2024-04-24T23:50:19.085385+0000","last_became_peered":"2024-04-24T23:50:19.085385+0000","last_unstale":"2024-04-24T23:50:20.787850+0000","last_undegraded":"2024-04-24T23:50:20.787850+0000","last_fullsized":"2024-04-24T23:50:20.787850+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:58.003442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.789200+0000","last_change":"2024-04-24T23:50:19.089120+0000","last_active":"2024-04-24T23:50:20.789200+0000","last_peered":"2024-04-24T23:50:20.789200+0000","last_clean":"2024-04-24T23:50:20.789200+0000","last_became_active":"2024-04-24T23:50:19.088862+0000","last_became_peered":"2024-04-24T23:50:19.088862+0000","last_unstale":"2024-04-24T23:50:20.789200+0000","last_undegraded":"2024-04-24T23:50:20.789200+0000","last_fullsized":"2024-04-24T23:50:20.789200+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:54:49.964613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788361+0000","last_change":"2024-04-24T23:50:19.079394+0000","last_active":"2024-04-24T23:50:20.788361+0000","last_peered":"2024-04-24T23:50:20.788361+0000","last_clean":"2024-04-24T23:50:20.788361+0000","last_became_active":"2024-04-24T23:50:19.079012+0000","last_became_peered":"2024-04-24T23:50:19.079012+0000","last_unstale":"2024-04-24T23:50:20.788361+0000","last_undegraded":"2024-04-24T23:50:20.788361+0000","last_fullsized":"2024-04-24T23:50:20.788361+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:16:23.638767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788469+0000","last_change":"2024-04-24T23:50:19.091815+0000","last_active":"2024-04-24T23:50:20.788469+0000","last_peered":"2024-04-24T23:50:20.788469+0000","last_clean":"2024-04-24T23:50:20.788469+0000","last_became_active":"2024-04-24T23:50:19.091486+0000","last_became_peered":"2024-04-24T23:50:19.091486+0000","last_unstale":"2024-04-24T23:50:20.788469+0000","last_undegraded":"2024-04-24T23:50:20.788469+0000","last_fullsized":"2024-04-24T23:50:20.788469+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:58.858220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788075+0000","last_change":"2024-04-24T23:50:19.086585+0000","last_active":"2024-04-24T23:50:20.788075+0000","last_peered":"2024-04-24T23:50:20.788075+0000","last_clean":"2024-04-24T23:50:20.788075+0000","last_became_active":"2024-04-24T23:50:19.086325+0000","last_became_peered":"2024-04-24T23:50:19.086325+0000","last_unstale":"2024-04-24T23:50:20.788075+0000","last_undegraded":"2024-04-24T23:50:20.788075+0000","last_fullsized":"2024-04-24T23:50:20.788075+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:03.073135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788110+0000","last_change":"2024-04-24T23:50:19.079371+0000","last_active":"2024-04-24T23:50:20.788110+0000","last_peered":"2024-04-24T23:50:20.788110+0000","last_clean":"2024-04-24T23:50:20.788110+0000","last_became_active":"2024-04-24T23:50:19.079080+0000","last_became_peered":"2024-04-24T23:50:19.079080+0000","last_unstale":"2024-04-24T23:50:20.788110+0000","last_undegraded":"2024-04-24T23:50:20.788110+0000","last_fullsized":"2024-04-24T23:50:20.788110+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:00.216226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075692+0000","last_change":"2024-04-24T23:50:19.086958+0000","last_active":"2024-04-24T23:50:20.075692+0000","last_peered":"2024-04-24T23:50:20.075692+0000","last_clean":"2024-04-24T23:50:20.075692+0000","last_became_active":"2024-04-24T23:50:19.086664+0000","last_became_peered":"2024-04-24T23:50:19.086664+0000","last_unstale":"2024-04-24T23:50:20.075692+0000","last_undegraded":"2024-04-24T23:50:20.075692+0000","last_fullsized":"2024-04-24T23:50:20.075692+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:03.073135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788040+0000","last_change":"2024-04-24T23:50:19.089302+0000","last_active":"2024-04-24T23:50:20.788040+0000","last_peered":"2024-04-24T23:50:20.788040+0000","last_clean":"2024-04-24T23:50:20.788040+0000","last_became_active":"2024-04-24T23:50:19.086389+0000","last_became_peered":"2024-04-24T23:50:19.086389+0000","last_unstale":"2024-04-24T23:50:20.788040+0000","last_undegraded":"2024-04-24T23:50:20.788040+0000","last_fullsized":"2024-04-24T23:50:20.788040+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:44.423218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074130+0000","last_change":"2024-04-24T23:50:19.083004+0000","last_active":"2024-04-24T23:50:20.074130+0000","last_peered":"2024-04-24T23:50:20.074130+0000","last_clean":"2024-04-24T23:50:20.074130+0000","last_became_active":"2024-04-24T23:50:19.082723+0000","last_became_peered":"2024-04-24T23:50:19.082723+0000","last_unstale":"2024-04-24T23:50:20.074130+0000","last_undegraded":"2024-04-24T23:50:20.074130+0000","last_fullsized":"2024-04-24T23:50:20.074130+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:46.624588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075281+0000","last_change":"2024-04-24T23:50:19.082525+0000","last_active":"2024-04-24T23:50:20.075281+0000","last_peered":"2024-04-24T23:50:20.075281+0000","last_clean":"2024-04-24T23:50:20.075281+0000","last_became_active":"2024-04-24T23:50:19.081187+0000","last_became_peered":"2024-04-24T23:50:19.081187+0000","last_unstale":"2024-04-24T23:50:20.075281+0000","last_undegraded":"2024-04-24T23:50:20.075281+0000","last_fullsized":"2024-04-24T23:50:20.075281+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:42:41.658151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073673+0000","last_change":"2024-04-24T23:50:19.089420+0000","last_active":"2024-04-24T23:50:20.073673+0000","last_peered":"2024-04-24T23:50:20.073673+0000","last_clean":"2024-04-24T23:50:20.073673+0000","last_became_active":"2024-04-24T23:50:19.089197+0000","last_became_peered":"2024-04-24T23:50:19.089197+0000","last_unstale":"2024-04-24T23:50:20.073673+0000","last_undegraded":"2024-04-24T23:50:20.073673+0000","last_fullsized":"2024-04-24T23:50:20.073673+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:50.774708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788464+0000","last_change":"2024-04-24T23:50:19.086234+0000","last_active":"2024-04-24T23:50:20.788464+0000","last_peered":"2024-04-24T23:50:20.788464+0000","last_clean":"2024-04-24T23:50:20.788464+0000","last_became_active":"2024-04-24T23:50:19.085987+0000","last_became_peered":"2024-04-24T23:50:19.085987+0000","last_unstale":"2024-04-24T23:50:20.788464+0000","last_undegraded":"2024-04-24T23:50:20.788464+0000","last_fullsized":"2024-04-24T23:50:20.788464+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:04.482202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073441+0000","last_change":"2024-04-24T23:50:19.090305+0000","last_active":"2024-04-24T23:50:20.073441+0000","last_peered":"2024-04-24T23:50:20.073441+0000","last_clean":"2024-04-24T23:50:20.073441+0000","last_became_active":"2024-04-24T23:50:19.090023+0000","last_became_peered":"2024-04-24T23:50:19.090023+0000","last_unstale":"2024-04-24T23:50:20.073441+0000","last_undegraded":"2024-04-24T23:50:20.073441+0000","last_fullsized":"2024-04-24T23:50:20.073441+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:56.437091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788183+0000","last_change":"2024-04-24T23:50:19.086653+0000","last_active":"2024-04-24T23:50:20.788183+0000","last_peered":"2024-04-24T23:50:20.788183+0000","last_clean":"2024-04-24T23:50:20.788183+0000","last_became_active":"2024-04-24T23:50:19.086376+0000","last_became_peered":"2024-04-24T23:50:19.086376+0000","last_unstale":"2024-04-24T23:50:20.788183+0000","last_undegraded":"2024-04-24T23:50:20.788183+0000","last_fullsized":"2024-04-24T23:50:20.788183+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:59.007356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.076072+0000","last_change":"2024-04-24T23:50:19.093829+0000","last_active":"2024-04-24T23:50:20.076072+0000","last_peered":"2024-04-24T23:50:20.076072+0000","last_clean":"2024-04-24T23:50:20.076072+0000","last_became_active":"2024-04-24T23:50:19.092122+0000","last_became_peered":"2024-04-24T23:50:19.092122+0000","last_unstale":"2024-04-24T23:50:20.076072+0000","last_undegraded":"2024-04-24T23:50:20.076072+0000","last_fullsized":"2024-04-24T23:50:20.076072+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:59.514575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074752+0000","last_change":"2024-04-24T23:50:19.079759+0000","last_active":"2024-04-24T23:50:20.074752+0000","last_peered":"2024-04-24T23:50:20.074752+0000","last_clean":"2024-04-24T23:50:20.074752+0000","last_became_active":"2024-04-24T23:50:19.079496+0000","last_became_peered":"2024-04-24T23:50:19.079496+0000","last_unstale":"2024-04-24T23:50:20.074752+0000","last_undegraded":"2024-04-24T23:50:20.074752+0000","last_fullsized":"2024-04-24T23:50:20.074752+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:09.036785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788031+0000","last_change":"2024-04-24T23:50:19.077391+0000","last_active":"2024-04-24T23:50:20.788031+0000","last_peered":"2024-04-24T23:50:20.788031+0000","last_clean":"2024-04-24T23:50:20.788031+0000","last_became_active":"2024-04-24T23:50:19.077099+0000","last_became_peered":"2024-04-24T23:50:19.077099+0000","last_unstale":"2024-04-24T23:50:20.788031+0000","last_undegraded":"2024-04-24T23:50:20.788031+0000","last_fullsized":"2024-04-24T23:50:20.788031+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:55:28.732222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073952+0000","last_change":"2024-04-24T23:50:19.089032+0000","last_active":"2024-04-24T23:50:20.073952+0000","last_peered":"2024-04-24T23:50:20.073952+0000","last_clean":"2024-04-24T23:50:20.073952+0000","last_became_active":"2024-04-24T23:50:19.088739+0000","last_became_peered":"2024-04-24T23:50:19.088739+0000","last_unstale":"2024-04-24T23:50:20.073952+0000","last_undegraded":"2024-04-24T23:50:20.073952+0000","last_fullsized":"2024-04-24T23:50:20.073952+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:39:01.832638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.072104+0000","last_change":"2024-04-24T23:50:19.083395+0000","last_active":"2024-04-24T23:50:20.072104+0000","last_peered":"2024-04-24T23:50:20.072104+0000","last_clean":"2024-04-24T23:50:20.072104+0000","last_became_active":"2024-04-24T23:50:19.083139+0000","last_became_peered":"2024-04-24T23:50:19.083139+0000","last_unstale":"2024-04-24T23:50:20.072104+0000","last_undegraded":"2024-04-24T23:50:20.072104+0000","last_fullsized":"2024-04-24T23:50:20.072104+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:54:49.964613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073804+0000","last_change":"2024-04-24T23:50:19.081999+0000","last_active":"2024-04-24T23:50:20.073804+0000","last_peered":"2024-04-24T23:50:20.073804+0000","last_clean":"2024-04-24T23:50:20.073804+0000","last_became_active":"2024-04-24T23:50:19.081681+0000","last_became_peered":"2024-04-24T23:50:19.081681+0000","last_unstale":"2024-04-24T23:50:20.073804+0000","last_undegraded":"2024-04-24T23:50:20.073804+0000","last_fullsized":"2024-04-24T23:50:20.073804+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:03.073135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788421+0000","last_change":"2024-04-24T23:50:19.091112+0000","last_active":"2024-04-24T23:50:20.788421+0000","last_peered":"2024-04-24T23:50:20.788421+0000","last_clean":"2024-04-24T23:50:20.788421+0000","last_became_active":"2024-04-24T23:50:19.090810+0000","last_became_peered":"2024-04-24T23:50:19.090810+0000","last_unstale":"2024-04-24T23:50:20.788421+0000","last_undegraded":"2024-04-24T23:50:20.788421+0000","last_fullsized":"2024-04-24T23:50:20.788421+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:26.687147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074013+0000","last_change":"2024-04-24T23:50:19.093267+0000","last_active":"2024-04-24T23:50:20.074013+0000","last_peered":"2024-04-24T23:50:20.074013+0000","last_clean":"2024-04-24T23:50:20.074013+0000","last_became_active":"2024-04-24T23:50:19.092999+0000","last_became_peered":"2024-04-24T23:50:19.092999+0000","last_unstale":"2024-04-24T23:50:20.074013+0000","last_undegraded":"2024-04-24T23:50:20.074013+0000","last_fullsized":"2024-04-24T23:50:20.074013+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:39.114823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074577+0000","last_change":"2024-04-24T23:50:19.083414+0000","last_active":"2024-04-24T23:50:20.074577+0000","last_peered":"2024-04-24T23:50:20.074577+0000","last_clean":"2024-04-24T23:50:20.074577+0000","last_became_active":"2024-04-24T23:50:19.083007+0000","last_became_peered":"2024-04-24T23:50:19.083007+0000","last_unstale":"2024-04-24T23:50:20.074577+0000","last_undegraded":"2024-04-24T23:50:20.074577+0000","last_fullsized":"2024-04-24T23:50:20.074577+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:59.514575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074759+0000","last_change":"2024-04-24T23:50:19.079184+0000","last_active":"2024-04-24T23:50:20.074759+0000","last_peered":"2024-04-24T23:50:20.074759+0000","last_clean":"2024-04-24T23:50:20.074759+0000","last_became_active":"2024-04-24T23:50:19.078943+0000","last_became_peered":"2024-04-24T23:50:19.078943+0000","last_unstale":"2024-04-24T23:50:20.074759+0000","last_undegraded":"2024-04-24T23:50:20.074759+0000","last_fullsized":"2024-04-24T23:50:20.074759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:04.875514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.072053+0000","last_change":"2024-04-24T23:50:19.084875+0000","last_active":"2024-04-24T23:50:20.072053+0000","last_peered":"2024-04-24T23:50:20.072053+0000","last_clean":"2024-04-24T23:50:20.072053+0000","last_became_active":"2024-04-24T23:50:19.084583+0000","last_became_peered":"2024-04-24T23:50:19.084583+0000","last_unstale":"2024-04-24T23:50:20.072053+0000","last_undegraded":"2024-04-24T23:50:20.072053+0000","last_fullsized":"2024-04-24T23:50:20.072053+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:53.415628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075059+0000","last_change":"2024-04-24T23:50:19.077387+0000","last_active":"2024-04-24T23:50:20.075059+0000","last_peered":"2024-04-24T23:50:20.075059+0000","last_clean":"2024-04-24T23:50:20.075059+0000","last_became_active":"2024-04-24T23:50:19.077072+0000","last_became_peered":"2024-04-24T23:50:19.077072+0000","last_unstale":"2024-04-24T23:50:20.075059+0000","last_undegraded":"2024-04-24T23:50:20.075059+0000","last_fullsized":"2024-04-24T23:50:20.075059+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:17.465127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073963+0000","last_change":"2024-04-24T23:50:18.085951+0000","last_active":"2024-04-24T23:50:20.073963+0000","last_peered":"2024-04-24T23:50:20.073963+0000","last_clean":"2024-04-24T23:50:20.073963+0000","last_became_active":"2024-04-24T23:50:18.085687+0000","last_became_peered":"2024-04-24T23:50:18.085687+0000","last_unstale":"2024-04-24T23:50:20.073963+0000","last_undegraded":"2024-04-24T23:50:20.073963+0000","last_fullsized":"2024-04-24T23:50:20.073963+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:36:44.740515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.072032+0000","last_change":"2024-04-24T23:50:19.084128+0000","last_active":"2024-04-24T23:50:20.072032+0000","last_peered":"2024-04-24T23:50:20.072032+0000","last_clean":"2024-04-24T23:50:20.072032+0000","last_became_active":"2024-04-24T23:50:19.083841+0000","last_became_peered":"2024-04-24T23:50:19.083841+0000","last_unstale":"2024-04-24T23:50:20.072032+0000","last_undegraded":"2024-04-24T23:50:20.072032+0000","last_fullsized":"2024-04-24T23:50:20.072032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:20.973639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788569+0000","last_change":"2024-04-24T23:50:18.086468+0000","last_active":"2024-04-24T23:50:20.788569+0000","last_peered":"2024-04-24T23:50:20.788569+0000","last_clean":"2024-04-24T23:50:20.788569+0000","last_became_active":"2024-04-24T23:50:18.086193+0000","last_became_peered":"2024-04-24T23:50:18.086193+0000","last_unstale":"2024-04-24T23:50:20.788569+0000","last_undegraded":"2024-04-24T23:50:20.788569+0000","last_fullsized":"2024-04-24T23:50:20.788569+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:36:44.740515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.076139+0000","last_change":"2024-04-24T23:50:19.088008+0000","last_active":"2024-04-24T23:50:20.076139+0000","last_peered":"2024-04-24T23:50:20.076139+0000","last_clean":"2024-04-24T23:50:20.076139+0000","last_became_active":"2024-04-24T23:50:19.087619+0000","last_became_peered":"2024-04-24T23:50:19.087619+0000","last_unstale":"2024-04-24T23:50:20.076139+0000","last_undegraded":"2024-04-24T23:50:20.076139+0000","last_fullsized":"2024-04-24T23:50:20.076139+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:10.966605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.072081+0000","last_change":"2024-04-24T23:50:18.084764+0000","last_active":"2024-04-24T23:50:20.072081+0000","last_peered":"2024-04-24T23:50:20.072081+0000","last_clean":"2024-04-24T23:50:20.072081+0000","last_became_active":"2024-04-24T23:50:18.084197+0000","last_became_peered":"2024-04-24T23:50:18.084197+0000","last_unstale":"2024-04-24T23:50:20.072081+0000","last_undegraded":"2024-04-24T23:50:20.072081+0000","last_fullsized":"2024-04-24T23:50:20.072081+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:23:27.985620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788257+0000","last_change":"2024-04-24T23:50:19.087571+0000","last_active":"2024-04-24T23:50:20.788257+0000","last_peered":"2024-04-24T23:50:20.788257+0000","last_clean":"2024-04-24T23:50:20.788257+0000","last_became_active":"2024-04-24T23:50:19.087171+0000","last_became_peered":"2024-04-24T23:50:19.087171+0000","last_unstale":"2024-04-24T23:50:20.788257+0000","last_undegraded":"2024-04-24T23:50:20.788257+0000","last_fullsized":"2024-04-24T23:50:20.788257+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:00.216226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075985+0000","last_change":"2024-04-24T23:50:18.088963+0000","last_active":"2024-04-24T23:50:20.075985+0000","last_peered":"2024-04-24T23:50:20.075985+0000","last_clean":"2024-04-24T23:50:20.075985+0000","last_became_active":"2024-04-24T23:50:18.088648+0000","last_became_peered":"2024-04-24T23:50:18.088648+0000","last_unstale":"2024-04-24T23:50:20.075985+0000","last_undegraded":"2024-04-24T23:50:20.075985+0000","last_fullsized":"2024-04-24T23:50:20.075985+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:19:42.318258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788657+0000","last_change":"2024-04-24T23:50:19.083448+0000","last_active":"2024-04-24T23:50:20.788657+0000","last_peered":"2024-04-24T23:50:20.788657+0000","last_clean":"2024-04-24T23:50:20.788657+0000","last_became_active":"2024-04-24T23:50:19.083197+0000","last_became_peered":"2024-04-24T23:50:19.083197+0000","last_unstale":"2024-04-24T23:50:20.788657+0000","last_undegraded":"2024-04-24T23:50:20.788657+0000","last_fullsized":"2024-04-24T23:50:20.788657+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:59.514575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788199+0000","last_change":"2024-04-24T23:50:18.088565+0000","last_active":"2024-04-24T23:50:20.788199+0000","last_peered":"2024-04-24T23:50:20.788199+0000","last_clean":"2024-04-24T23:50:20.788199+0000","last_became_active":"2024-04-24T23:50:18.087754+0000","last_became_peered":"2024-04-24T23:50:18.087754+0000","last_unstale":"2024-04-24T23:50:20.788199+0000","last_undegraded":"2024-04-24T23:50:20.788199+0000","last_fullsized":"2024-04-24T23:50:20.788199+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:49.773026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788207+0000","last_change":"2024-04-24T23:50:19.089083+0000","last_active":"2024-04-24T23:50:20.788207+0000","last_peered":"2024-04-24T23:50:20.788207+0000","last_clean":"2024-04-24T23:50:20.788207+0000","last_became_active":"2024-04-24T23:50:19.086994+0000","last_became_peered":"2024-04-24T23:50:19.086994+0000","last_unstale":"2024-04-24T23:50:20.788207+0000","last_undegraded":"2024-04-24T23:50:20.788207+0000","last_fullsized":"2024-04-24T23:50:20.788207+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:53.860893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073105+0000","last_change":"2024-04-24T23:50:18.084696+0000","last_active":"2024-04-24T23:50:20.073105+0000","last_peered":"2024-04-24T23:50:20.073105+0000","last_clean":"2024-04-24T23:50:20.073105+0000","last_became_active":"2024-04-24T23:50:18.084421+0000","last_became_peered":"2024-04-24T23:50:18.084421+0000","last_unstale":"2024-04-24T23:50:20.073105+0000","last_undegraded":"2024-04-24T23:50:20.073105+0000","last_fullsized":"2024-04-24T23:50:20.073105+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:09:52.280197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075868+0000","last_change":"2024-04-24T23:50:19.092620+0000","last_active":"2024-04-24T23:50:20.075868+0000","last_peered":"2024-04-24T23:50:20.075868+0000","last_clean":"2024-04-24T23:50:20.075868+0000","last_became_active":"2024-04-24T23:50:19.092166+0000","last_became_peered":"2024-04-24T23:50:19.092166+0000","last_unstale":"2024-04-24T23:50:20.075868+0000","last_undegraded":"2024-04-24T23:50:20.075868+0000","last_fullsized":"2024-04-24T23:50:20.075868+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:39.114823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.071743+0000","last_change":"2024-04-24T23:50:18.088633+0000","last_active":"2024-04-24T23:50:20.071743+0000","last_peered":"2024-04-24T23:50:20.071743+0000","last_clean":"2024-04-24T23:50:20.071743+0000","last_became_active":"2024-04-24T23:50:18.088327+0000","last_became_peered":"2024-04-24T23:50:18.088327+0000","last_unstale":"2024-04-24T23:50:20.071743+0000","last_undegraded":"2024-04-24T23:50:20.071743+0000","last_fullsized":"2024-04-24T23:50:20.071743+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:09:52.280197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788141+0000","last_change":"2024-04-24T23:50:19.096980+0000","last_active":"2024-04-24T23:50:20.788141+0000","last_peered":"2024-04-24T23:50:20.788141+0000","last_clean":"2024-04-24T23:50:20.788141+0000","last_became_active":"2024-04-24T23:50:19.096697+0000","last_became_peered":"2024-04-24T23:50:19.096697+0000","last_unstale":"2024-04-24T23:50:20.788141+0000","last_undegraded":"2024-04-24T23:50:20.788141+0000","last_fullsized":"2024-04-24T23:50:20.788141+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:49:15.545291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075551+0000","last_change":"2024-04-24T23:50:18.084321+0000","last_active":"2024-04-24T23:50:20.075551+0000","last_peered":"2024-04-24T23:50:20.075551+0000","last_clean":"2024-04-24T23:50:20.075551+0000","last_became_active":"2024-04-24T23:50:18.084049+0000","last_became_peered":"2024-04-24T23:50:18.084049+0000","last_unstale":"2024-04-24T23:50:20.075551+0000","last_undegraded":"2024-04-24T23:50:20.075551+0000","last_fullsized":"2024-04-24T23:50:20.075551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:40.483306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788855+0000","last_change":"2024-04-24T23:50:19.090430+0000","last_active":"2024-04-24T23:50:20.788855+0000","last_peered":"2024-04-24T23:50:20.788855+0000","last_clean":"2024-04-24T23:50:20.788855+0000","last_became_active":"2024-04-24T23:50:19.089972+0000","last_became_peered":"2024-04-24T23:50:19.089972+0000","last_unstale":"2024-04-24T23:50:20.788855+0000","last_undegraded":"2024-04-24T23:50:20.788855+0000","last_fullsized":"2024-04-24T23:50:20.788855+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:26:35.411631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788447+0000","last_change":"2024-04-24T23:50:18.090464+0000","last_active":"2024-04-24T23:50:20.788447+0000","last_peered":"2024-04-24T23:50:20.788447+0000","last_clean":"2024-04-24T23:50:20.788447+0000","last_became_active":"2024-04-24T23:50:18.090177+0000","last_became_peered":"2024-04-24T23:50:18.090177+0000","last_unstale":"2024-04-24T23:50:20.788447+0000","last_undegraded":"2024-04-24T23:50:20.788447+0000","last_fullsized":"2024-04-24T23:50:20.788447+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:02.071453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075163+0000","last_change":"2024-04-24T23:50:19.084335+0000","last_active":"2024-04-24T23:50:20.075163+0000","last_peered":"2024-04-24T23:50:20.075163+0000","last_clean":"2024-04-24T23:50:20.075163+0000","last_became_active":"2024-04-24T23:50:19.084050+0000","last_became_peered":"2024-04-24T23:50:19.084050+0000","last_unstale":"2024-04-24T23:50:20.075163+0000","last_undegraded":"2024-04-24T23:50:20.075163+0000","last_fullsized":"2024-04-24T23:50:20.075163+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:49:15.545291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788379+0000","last_change":"2024-04-24T23:50:18.086806+0000","last_active":"2024-04-24T23:50:20.788379+0000","last_peered":"2024-04-24T23:50:20.788379+0000","last_clean":"2024-04-24T23:50:20.788379+0000","last_became_active":"2024-04-24T23:50:18.086489+0000","last_became_peered":"2024-04-24T23:50:18.086489+0000","last_unstale":"2024-04-24T23:50:20.788379+0000","last_undegraded":"2024-04-24T23:50:20.788379+0000","last_fullsized":"2024-04-24T23:50:20.788379+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:02.071453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788513+0000","last_change":"2024-04-24T23:50:19.091664+0000","last_active":"2024-04-24T23:50:20.788513+0000","last_peered":"2024-04-24T23:50:20.788513+0000","last_clean":"2024-04-24T23:50:20.788513+0000","last_became_active":"2024-04-24T23:50:19.090350+0000","last_became_peered":"2024-04-24T23:50:19.090350+0000","last_unstale":"2024-04-24T23:50:20.788513+0000","last_undegraded":"2024-04-24T23:50:20.788513+0000","last_fullsized":"2024-04-24T23:50:20.788513+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:39.114823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788516+0000","last_change":"2024-04-24T23:50:18.081470+0000","last_active":"2024-04-24T23:50:20.788516+0000","last_peered":"2024-04-24T23:50:20.788516+0000","last_clean":"2024-04-24T23:50:20.788516+0000","last_became_active":"2024-04-24T23:50:18.081021+0000","last_became_peered":"2024-04-24T23:50:18.081021+0000","last_unstale":"2024-04-24T23:50:20.788516+0000","last_undegraded":"2024-04-24T23:50:20.788516+0000","last_fullsized":"2024-04-24T23:50:20.788516+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:38.113141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.076180+0000","last_change":"2024-04-24T23:50:19.089223+0000","last_active":"2024-04-24T23:50:20.076180+0000","last_peered":"2024-04-24T23:50:20.076180+0000","last_clean":"2024-04-24T23:50:20.076180+0000","last_became_active":"2024-04-24T23:50:19.088972+0000","last_became_peered":"2024-04-24T23:50:19.088972+0000","last_unstale":"2024-04-24T23:50:20.076180+0000","last_undegraded":"2024-04-24T23:50:20.076180+0000","last_fullsized":"2024-04-24T23:50:20.076180+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:53.415628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788830+0000","last_change":"2024-04-24T23:50:18.086865+0000","last_active":"2024-04-24T23:50:20.788830+0000","last_peered":"2024-04-24T23:50:20.788830+0000","last_clean":"2024-04-24T23:50:20.788830+0000","last_became_active":"2024-04-24T23:50:18.086448+0000","last_became_peered":"2024-04-24T23:50:18.086448+0000","last_unstale":"2024-04-24T23:50:20.788830+0000","last_undegraded":"2024-04-24T23:50:20.788830+0000","last_fullsized":"2024-04-24T23:50:20.788830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:49.773026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788820+0000","last_change":"2024-04-24T23:50:19.088463+0000","last_active":"2024-04-24T23:50:20.788820+0000","last_peered":"2024-04-24T23:50:20.788820+0000","last_clean":"2024-04-24T23:50:20.788820+0000","last_became_active":"2024-04-24T23:50:19.088206+0000","last_became_peered":"2024-04-24T23:50:19.088206+0000","last_unstale":"2024-04-24T23:50:20.788820+0000","last_undegraded":"2024-04-24T23:50:20.788820+0000","last_fullsized":"2024-04-24T23:50:20.788820+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:10.966605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788258+0000","last_change":"2024-04-24T23:50:18.087815+0000","last_active":"2024-04-24T23:50:20.788258+0000","last_peered":"2024-04-24T23:50:20.788258+0000","last_clean":"2024-04-24T23:50:20.788258+0000","last_became_active":"2024-04-24T23:50:18.087511+0000","last_became_peered":"2024-04-24T23:50:18.087511+0000","last_unstale":"2024-04-24T23:50:20.788258+0000","last_undegraded":"2024-04-24T23:50:20.788258+0000","last_fullsized":"2024-04-24T23:50:20.788258+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:55.435409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.787972+0000","last_change":"2024-04-24T23:50:19.090999+0000","last_active":"2024-04-24T23:50:20.787972+0000","last_peered":"2024-04-24T23:50:20.787972+0000","last_clean":"2024-04-24T23:50:20.787972+0000","last_became_active":"2024-04-24T23:50:19.090687+0000","last_became_peered":"2024-04-24T23:50:19.090687+0000","last_unstale":"2024-04-24T23:50:20.787972+0000","last_undegraded":"2024-04-24T23:50:20.787972+0000","last_fullsized":"2024-04-24T23:50:20.787972+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:17.465127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.071872+0000","last_change":"2024-04-24T23:50:18.087633+0000","last_active":"2024-04-24T23:50:20.071872+0000","last_peered":"2024-04-24T23:50:20.071872+0000","last_clean":"2024-04-24T23:50:20.071872+0000","last_became_active":"2024-04-24T23:50:18.087254+0000","last_became_peered":"2024-04-24T23:50:18.087254+0000","last_unstale":"2024-04-24T23:50:20.071872+0000","last_undegraded":"2024-04-24T23:50:20.071872+0000","last_fullsized":"2024-04-24T23:50:20.071872+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:55:08.953626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073039+0000","last_change":"2024-04-24T23:50:19.078886+0000","last_active":"2024-04-24T23:50:20.073039+0000","last_peered":"2024-04-24T23:50:20.073039+0000","last_clean":"2024-04-24T23:50:20.073039+0000","last_became_active":"2024-04-24T23:50:19.078610+0000","last_became_peered":"2024-04-24T23:50:19.078610+0000","last_unstale":"2024-04-24T23:50:20.073039+0000","last_undegraded":"2024-04-24T23:50:20.073039+0000","last_fullsized":"2024-04-24T23:50:20.073039+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:54:49.964613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073370+0000","last_change":"2024-04-24T23:50:18.084874+0000","last_active":"2024-04-24T23:50:20.073370+0000","last_peered":"2024-04-24T23:50:20.073370+0000","last_clean":"2024-04-24T23:50:20.073370+0000","last_became_active":"2024-04-24T23:50:18.083917+0000","last_became_peered":"2024-04-24T23:50:18.083917+0000","last_unstale":"2024-04-24T23:50:20.073370+0000","last_undegraded":"2024-04-24T23:50:20.073370+0000","last_fullsized":"2024-04-24T23:50:20.073370+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:40.483306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788197+0000","last_change":"2024-04-24T23:50:19.091744+0000","last_active":"2024-04-24T23:50:20.788197+0000","last_peered":"2024-04-24T23:50:20.788197+0000","last_clean":"2024-04-24T23:50:20.788197+0000","last_became_active":"2024-04-24T23:50:19.091084+0000","last_became_peered":"2024-04-24T23:50:19.091084+0000","last_unstale":"2024-04-24T23:50:20.788197+0000","last_undegraded":"2024-04-24T23:50:20.788197+0000","last_fullsized":"2024-04-24T23:50:20.788197+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:42:41.658151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074196+0000","last_change":"2024-04-24T23:50:18.085038+0000","last_active":"2024-04-24T23:50:20.074196+0000","last_peered":"2024-04-24T23:50:20.074196+0000","last_clean":"2024-04-24T23:50:20.074196+0000","last_became_active":"2024-04-24T23:50:18.084787+0000","last_became_peered":"2024-04-24T23:50:18.084787+0000","last_unstale":"2024-04-24T23:50:20.074196+0000","last_undegraded":"2024-04-24T23:50:20.074196+0000","last_fullsized":"2024-04-24T23:50:20.074196+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:30.330350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788648+0000","last_change":"2024-04-24T23:50:19.086313+0000","last_active":"2024-04-24T23:50:20.788648+0000","last_peered":"2024-04-24T23:50:20.788648+0000","last_clean":"2024-04-24T23:50:20.788648+0000","last_became_active":"2024-04-24T23:50:19.085565+0000","last_became_peered":"2024-04-24T23:50:19.085565+0000","last_unstale":"2024-04-24T23:50:20.788648+0000","last_undegraded":"2024-04-24T23:50:20.788648+0000","last_fullsized":"2024-04-24T23:50:20.788648+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:19.118407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788318+0000","last_change":"2024-04-24T23:50:18.081636+0000","last_active":"2024-04-24T23:50:20.788318+0000","last_peered":"2024-04-24T23:50:20.788318+0000","last_clean":"2024-04-24T23:50:20.788318+0000","last_became_active":"2024-04-24T23:50:18.081311+0000","last_became_peered":"2024-04-24T23:50:18.081311+0000","last_unstale":"2024-04-24T23:50:20.788318+0000","last_undegraded":"2024-04-24T23:50:20.788318+0000","last_fullsized":"2024-04-24T23:50:20.788318+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:55.435409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788468+0000","last_change":"2024-04-24T23:50:19.087011+0000","last_active":"2024-04-24T23:50:20.788468+0000","last_peered":"2024-04-24T23:50:20.788468+0000","last_clean":"2024-04-24T23:50:20.788468+0000","last_became_active":"2024-04-24T23:50:19.086515+0000","last_became_peered":"2024-04-24T23:50:19.086515+0000","last_unstale":"2024-04-24T23:50:20.788468+0000","last_undegraded":"2024-04-24T23:50:20.788468+0000","last_fullsized":"2024-04-24T23:50:20.788468+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:39:32.387993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.076195+0000","last_change":"2024-04-24T23:50:18.086714+0000","last_active":"2024-04-24T23:50:20.076195+0000","last_peered":"2024-04-24T23:50:20.076195+0000","last_clean":"2024-04-24T23:50:20.076195+0000","last_became_active":"2024-04-24T23:50:18.086282+0000","last_became_peered":"2024-04-24T23:50:18.086282+0000","last_unstale":"2024-04-24T23:50:20.076195+0000","last_undegraded":"2024-04-24T23:50:20.076195+0000","last_fullsized":"2024-04-24T23:50:20.076195+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:36:44.740515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.076131+0000","last_change":"2024-04-24T23:50:19.095372+0000","last_active":"2024-04-24T23:50:20.076131+0000","last_peered":"2024-04-24T23:50:20.076131+0000","last_clean":"2024-04-24T23:50:20.076131+0000","last_became_active":"2024-04-24T23:50:19.095122+0000","last_became_peered":"2024-04-24T23:50:19.095122+0000","last_unstale":"2024-04-24T23:50:20.076131+0000","last_undegraded":"2024-04-24T23:50:20.076131+0000","last_fullsized":"2024-04-24T23:50:20.076131+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:20.973639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788717+0000","last_change":"2024-04-24T23:50:18.076361+0000","last_active":"2024-04-24T23:50:20.788717+0000","last_peered":"2024-04-24T23:50:20.788717+0000","last_clean":"2024-04-24T23:50:20.788717+0000","last_became_active":"2024-04-24T23:50:18.076076+0000","last_became_peered":"2024-04-24T23:50:18.076076+0000","last_unstale":"2024-04-24T23:50:20.788717+0000","last_undegraded":"2024-04-24T23:50:20.788717+0000","last_fullsized":"2024-04-24T23:50:20.788717+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:03:10.961797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788847+0000","last_change":"2024-04-24T23:50:19.088399+0000","last_active":"2024-04-24T23:50:20.788847+0000","last_peered":"2024-04-24T23:50:20.788847+0000","last_clean":"2024-04-24T23:50:20.788847+0000","last_became_active":"2024-04-24T23:50:19.088106+0000","last_became_peered":"2024-04-24T23:50:19.088106+0000","last_unstale":"2024-04-24T23:50:20.788847+0000","last_undegraded":"2024-04-24T23:50:20.788847+0000","last_fullsized":"2024-04-24T23:50:20.788847+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:27:27.500102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.071862+0000","last_change":"2024-04-24T23:50:18.082589+0000","last_active":"2024-04-24T23:50:20.071862+0000","last_peered":"2024-04-24T23:50:20.071862+0000","last_clean":"2024-04-24T23:50:20.071862+0000","last_became_active":"2024-04-24T23:50:18.082312+0000","last_became_peered":"2024-04-24T23:50:18.082312+0000","last_unstale":"2024-04-24T23:50:20.071862+0000","last_undegraded":"2024-04-24T23:50:20.071862+0000","last_fullsized":"2024-04-24T23:50:20.071862+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:30.330350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074347+0000","last_change":"2024-04-24T23:50:19.079370+0000","last_active":"2024-04-24T23:50:20.074347+0000","last_peered":"2024-04-24T23:50:20.074347+0000","last_clean":"2024-04-24T23:50:20.074347+0000","last_became_active":"2024-04-24T23:50:19.079115+0000","last_became_peered":"2024-04-24T23:50:19.079115+0000","last_unstale":"2024-04-24T23:50:20.074347+0000","last_undegraded":"2024-04-24T23:50:20.074347+0000","last_fullsized":"2024-04-24T23:50:20.074347+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:12:12.871447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788464+0000","last_change":"2024-04-24T23:50:18.083687+0000","last_active":"2024-04-24T23:50:20.788464+0000","last_peered":"2024-04-24T23:50:20.788464+0000","last_clean":"2024-04-24T23:50:20.788464+0000","last_became_active":"2024-04-24T23:50:18.083380+0000","last_became_peered":"2024-04-24T23:50:18.083380+0000","last_unstale":"2024-04-24T23:50:20.788464+0000","last_undegraded":"2024-04-24T23:50:20.788464+0000","last_fullsized":"2024-04-24T23:50:20.788464+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:01:56.058772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788866+0000","last_change":"2024-04-24T23:50:19.089951+0000","last_active":"2024-04-24T23:50:20.788866+0000","last_peered":"2024-04-24T23:50:20.788866+0000","last_clean":"2024-04-24T23:50:20.788866+0000","last_became_active":"2024-04-24T23:50:19.089663+0000","last_became_peered":"2024-04-24T23:50:19.089663+0000","last_unstale":"2024-04-24T23:50:20.788866+0000","last_undegraded":"2024-04-24T23:50:20.788866+0000","last_fullsized":"2024-04-24T23:50:20.788866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:20.973639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788341+0000","last_change":"2024-04-24T23:50:18.084068+0000","last_active":"2024-04-24T23:50:20.788341+0000","last_peered":"2024-04-24T23:50:20.788341+0000","last_clean":"2024-04-24T23:50:20.788341+0000","last_became_active":"2024-04-24T23:50:18.083571+0000","last_became_peered":"2024-04-24T23:50:18.083571+0000","last_unstale":"2024-04-24T23:50:20.788341+0000","last_undegraded":"2024-04-24T23:50:20.788341+0000","last_fullsized":"2024-04-24T23:50:20.788341+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:26.194711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788086+0000","last_change":"2024-04-24T23:50:19.083773+0000","last_active":"2024-04-24T23:50:20.788086+0000","last_peered":"2024-04-24T23:50:20.788086+0000","last_clean":"2024-04-24T23:50:20.788086+0000","last_became_active":"2024-04-24T23:50:19.083351+0000","last_became_peered":"2024-04-24T23:50:19.083351+0000","last_unstale":"2024-04-24T23:50:20.788086+0000","last_undegraded":"2024-04-24T23:50:20.788086+0000","last_fullsized":"2024-04-24T23:50:20.788086+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:42:41.658151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074811+0000","last_change":"2024-04-24T23:50:18.082274+0000","last_active":"2024-04-24T23:50:20.074811+0000","last_peered":"2024-04-24T23:50:20.074811+0000","last_clean":"2024-04-24T23:50:20.074811+0000","last_became_active":"2024-04-24T23:50:18.082002+0000","last_became_peered":"2024-04-24T23:50:18.082002+0000","last_unstale":"2024-04-24T23:50:20.074811+0000","last_undegraded":"2024-04-24T23:50:20.074811+0000","last_fullsized":"2024-04-24T23:50:20.074811+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:23:27.985620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788282+0000","last_change":"2024-04-24T23:50:19.092660+0000","last_active":"2024-04-24T23:50:20.788282+0000","last_peered":"2024-04-24T23:50:20.788282+0000","last_clean":"2024-04-24T23:50:20.788282+0000","last_became_active":"2024-04-24T23:50:19.091442+0000","last_became_peered":"2024-04-24T23:50:19.091442+0000","last_unstale":"2024-04-24T23:50:20.788282+0000","last_undegraded":"2024-04-24T23:50:20.788282+0000","last_fullsized":"2024-04-24T23:50:20.788282+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:53.860893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788113+0000","last_change":"2024-04-24T23:50:18.084654+0000","last_active":"2024-04-24T23:50:20.788113+0000","last_peered":"2024-04-24T23:50:20.788113+0000","last_clean":"2024-04-24T23:50:20.788113+0000","last_became_active":"2024-04-24T23:50:18.084408+0000","last_became_peered":"2024-04-24T23:50:18.084408+0000","last_unstale":"2024-04-24T23:50:20.788113+0000","last_undegraded":"2024-04-24T23:50:20.788113+0000","last_fullsized":"2024-04-24T23:50:20.788113+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:58.919581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.071936+0000","last_change":"2024-04-24T23:50:19.081808+0000","last_active":"2024-04-24T23:50:20.071936+0000","last_peered":"2024-04-24T23:50:20.071936+0000","last_clean":"2024-04-24T23:50:20.071936+0000","last_became_active":"2024-04-24T23:50:19.081538+0000","last_became_peered":"2024-04-24T23:50:19.081538+0000","last_unstale":"2024-04-24T23:50:20.071936+0000","last_undegraded":"2024-04-24T23:50:20.071936+0000","last_fullsized":"2024-04-24T23:50:20.071936+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:10.966605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788462+0000","last_change":"2024-04-24T23:50:18.089751+0000","last_active":"2024-04-24T23:50:20.788462+0000","last_peered":"2024-04-24T23:50:20.788462+0000","last_clean":"2024-04-24T23:50:20.788462+0000","last_became_active":"2024-04-24T23:50:18.089446+0000","last_became_peered":"2024-04-24T23:50:18.089446+0000","last_unstale":"2024-04-24T23:50:20.788462+0000","last_undegraded":"2024-04-24T23:50:20.788462+0000","last_fullsized":"2024-04-24T23:50:20.788462+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:39:00.830956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075154+0000","last_change":"2024-04-24T23:50:19.084843+0000","last_active":"2024-04-24T23:50:20.075154+0000","last_peered":"2024-04-24T23:50:20.075154+0000","last_clean":"2024-04-24T23:50:20.075154+0000","last_became_active":"2024-04-24T23:50:19.084536+0000","last_became_peered":"2024-04-24T23:50:19.084536+0000","last_unstale":"2024-04-24T23:50:20.075154+0000","last_undegraded":"2024-04-24T23:50:20.075154+0000","last_fullsized":"2024-04-24T23:50:20.075154+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:44.423218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788632+0000","last_change":"2024-04-24T23:50:18.077055+0000","last_active":"2024-04-24T23:50:20.788632+0000","last_peered":"2024-04-24T23:50:20.788632+0000","last_clean":"2024-04-24T23:50:20.788632+0000","last_became_active":"2024-04-24T23:50:18.076771+0000","last_became_peered":"2024-04-24T23:50:18.076771+0000","last_unstale":"2024-04-24T23:50:20.788632+0000","last_undegraded":"2024-04-24T23:50:20.788632+0000","last_fullsized":"2024-04-24T23:50:20.788632+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:13.502204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788342+0000","last_change":"2024-04-24T23:50:19.094990+0000","last_active":"2024-04-24T23:50:20.788342+0000","last_peered":"2024-04-24T23:50:20.788342+0000","last_clean":"2024-04-24T23:50:20.788342+0000","last_became_active":"2024-04-24T23:50:19.094677+0000","last_became_peered":"2024-04-24T23:50:19.094677+0000","last_unstale":"2024-04-24T23:50:20.788342+0000","last_undegraded":"2024-04-24T23:50:20.788342+0000","last_fullsized":"2024-04-24T23:50:20.788342+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:59.007356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788295+0000","last_change":"2024-04-24T23:50:18.080646+0000","last_active":"2024-04-24T23:50:20.788295+0000","last_peered":"2024-04-24T23:50:20.788295+0000","last_clean":"2024-04-24T23:50:20.788295+0000","last_became_active":"2024-04-24T23:50:18.080375+0000","last_became_peered":"2024-04-24T23:50:18.080375+0000","last_unstale":"2024-04-24T23:50:20.788295+0000","last_undegraded":"2024-04-24T23:50:20.788295+0000","last_fullsized":"2024-04-24T23:50:20.788295+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:40.483306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788879+0000","last_change":"2024-04-24T23:50:19.087874+0000","last_active":"2024-04-24T23:50:20.788879+0000","last_peered":"2024-04-24T23:50:20.788879+0000","last_clean":"2024-04-24T23:50:20.788879+0000","last_became_active":"2024-04-24T23:50:19.087578+0000","last_became_peered":"2024-04-24T23:50:19.087578+0000","last_unstale":"2024-04-24T23:50:20.788879+0000","last_undegraded":"2024-04-24T23:50:20.788879+0000","last_fullsized":"2024-04-24T23:50:20.788879+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:53.415628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.789202+0000","last_change":"2024-04-24T23:50:18.084424+0000","last_active":"2024-04-24T23:50:20.789202+0000","last_peered":"2024-04-24T23:50:20.789202+0000","last_clean":"2024-04-24T23:50:20.789202+0000","last_became_active":"2024-04-24T23:50:18.084119+0000","last_became_peered":"2024-04-24T23:50:18.084119+0000","last_unstale":"2024-04-24T23:50:20.789202+0000","last_undegraded":"2024-04-24T23:50:20.789202+0000","last_fullsized":"2024-04-24T23:50:20.789202+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:19:42.318258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.071766+0000","last_change":"2024-04-24T23:50:19.093033+0000","last_active":"2024-04-24T23:50:20.071766+0000","last_peered":"2024-04-24T23:50:20.071766+0000","last_clean":"2024-04-24T23:50:20.071766+0000","last_became_active":"2024-04-24T23:50:19.092771+0000","last_became_peered":"2024-04-24T23:50:19.092771+0000","last_unstale":"2024-04-24T23:50:20.071766+0000","last_undegraded":"2024-04-24T23:50:20.071766+0000","last_fullsized":"2024-04-24T23:50:20.071766+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:39.114823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074641+0000","last_change":"2024-04-24T23:50:18.088182+0000","last_active":"2024-04-24T23:50:20.074641+0000","last_peered":"2024-04-24T23:50:20.074641+0000","last_clean":"2024-04-24T23:50:20.074641+0000","last_became_active":"2024-04-24T23:50:18.087899+0000","last_became_peered":"2024-04-24T23:50:18.087899+0000","last_unstale":"2024-04-24T23:50:20.074641+0000","last_undegraded":"2024-04-24T23:50:20.074641+0000","last_fullsized":"2024-04-24T23:50:20.074641+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:19:42.318258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075771+0000","last_change":"2024-04-24T23:50:19.090504+0000","last_active":"2024-04-24T23:50:20.075771+0000","last_peered":"2024-04-24T23:50:20.075771+0000","last_clean":"2024-04-24T23:50:20.075771+0000","last_became_active":"2024-04-24T23:50:19.088678+0000","last_became_peered":"2024-04-24T23:50:19.088678+0000","last_unstale":"2024-04-24T23:50:20.075771+0000","last_undegraded":"2024-04-24T23:50:20.075771+0000","last_fullsized":"2024-04-24T23:50:20.075771+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:39:01.832638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075576+0000","last_change":"2024-04-24T23:50:18.088177+0000","last_active":"2024-04-24T23:50:20.075576+0000","last_peered":"2024-04-24T23:50:20.075576+0000","last_clean":"2024-04-24T23:50:20.075576+0000","last_became_active":"2024-04-24T23:50:18.087915+0000","last_became_peered":"2024-04-24T23:50:18.087915+0000","last_unstale":"2024-04-24T23:50:20.075576+0000","last_undegraded":"2024-04-24T23:50:20.075576+0000","last_fullsized":"2024-04-24T23:50:20.075576+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:26:34.409949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074979+0000","last_change":"2024-04-24T23:50:19.080121+0000","last_active":"2024-04-24T23:50:20.074979+0000","last_peered":"2024-04-24T23:50:20.074979+0000","last_clean":"2024-04-24T23:50:20.074979+0000","last_became_active":"2024-04-24T23:50:19.079833+0000","last_became_peered":"2024-04-24T23:50:19.079833+0000","last_unstale":"2024-04-24T23:50:20.074979+0000","last_undegraded":"2024-04-24T23:50:20.074979+0000","last_fullsized":"2024-04-24T23:50:20.074979+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:58.003442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.787914+0000","last_change":"2024-04-24T23:50:18.090442+0000","last_active":"2024-04-24T23:50:20.787914+0000","last_peered":"2024-04-24T23:50:20.787914+0000","last_clean":"2024-04-24T23:50:20.787914+0000","last_became_active":"2024-04-24T23:50:18.090120+0000","last_became_peered":"2024-04-24T23:50:18.090120+0000","last_unstale":"2024-04-24T23:50:20.787914+0000","last_undegraded":"2024-04-24T23:50:20.787914+0000","last_fullsized":"2024-04-24T23:50:20.787914+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:39:49.970108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788385+0000","last_change":"2024-04-24T23:50:19.092659+0000","last_active":"2024-04-24T23:50:20.788385+0000","last_peered":"2024-04-24T23:50:20.788385+0000","last_clean":"2024-04-24T23:50:20.788385+0000","last_became_active":"2024-04-24T23:50:19.092380+0000","last_became_peered":"2024-04-24T23:50:19.092380+0000","last_unstale":"2024-04-24T23:50:20.788385+0000","last_undegraded":"2024-04-24T23:50:20.788385+0000","last_fullsized":"2024-04-24T23:50:20.788385+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:19.118407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074996+0000","last_change":"2024-04-24T23:50:18.083990+0000","last_active":"2024-04-24T23:50:20.074996+0000","last_peered":"2024-04-24T23:50:20.074996+0000","last_clean":"2024-04-24T23:50:20.074996+0000","last_became_active":"2024-04-24T23:50:18.083695+0000","last_became_peered":"2024-04-24T23:50:18.083695+0000","last_unstale":"2024-04-24T23:50:20.074996+0000","last_undegraded":"2024-04-24T23:50:20.074996+0000","last_fullsized":"2024-04-24T23:50:20.074996+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:55:08.953626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":206,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074318+0000","last_change":"2024-04-24T23:48:45.082459+0000","last_active":"2024-04-24T23:50:20.074318+0000","last_peered":"2024-04-24T23:50:20.074318+0000","last_clean":"2024-04-24T23:50:20.074318+0000","last_became_active":"2024-04-24T23:48:45.081495+0000","last_became_peered":"2024-04-24T23:48:45.081495+0000","last_unstale":"2024-04-24T23:50:20.074318+0000","last_undegraded":"2024-04-24T23:50:20.074318+0000","last_fullsized":"2024-04-24T23:50:20.074318+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_clean_scrub_stamp":"2024-04-24T23:48:43.440993+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:53:20.832511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":786757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":195289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":458752,"data_stored":194819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":458752,"data_stored":195311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31636,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340204,"statfs":{"total":96636764160,"available":96604368896,"internally_reserved":0,"allocated":466944,"data_stored":196353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":194777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":194797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1048576,"data_stored":785109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:50:24.115 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:50:24.117 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:50:26.540 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:50:26.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.654+0000 7f0c6e33b640 1 Processor -- start 2024-04-24T23:50:26.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.654+0000 7f0c6e33b640 1 -- start start 2024-04-24T23:50:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.655+0000 7f0c6e33b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c680d16c0 0x7f0c681601f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.655+0000 7f0c6e33b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f0c680705b0 0x7f0c6806faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.655+0000 7f0c6e33b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c681607d0 con 0x7f0c6815bcd0 2024-04-24T23:50:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.655+0000 7f0c6e33b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f0c6806f660 con 0x7f0c680705b0 2024-04-24T23:50:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.655+0000 7f0c6e33b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0c6806ed30 con 0x7f0c680d16c0 2024-04-24T23:50:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.655+0000 7f0c67fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0c6815bcd0 0x7f0c680d0fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:51732/0 (socket says 172.21.15.16:51732) 2024-04-24T23:50:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.655+0000 7f0c67fff640 1 -- 172.21.15.16:0/2324343855 learned_addr learned my addr 172.21.15.16:0/2324343855 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.655+0000 7f0c6c8d8640 1 --2- 172.21.15.16:0/2324343855 >> v2:172.21.15.156:3300/0 conn(0x7f0c680705b0 0x7f0c6806faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.655+0000 7f0c66ffd640 1 -- 172.21.15.16:0/2324343855 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3959011931 0 0) 0x7f0c681607d0 con 0x7f0c6815bcd0 2024-04-24T23:50:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.655+0000 7f0c677fe640 1 --2- 172.21.15.16:0/2324343855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c680d16c0 0x7f0c681601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.655+0000 7f0c66ffd640 1 -- 172.21.15.16:0/2324343855 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0c50003520 con 0x7f0c6815bcd0 2024-04-24T23:50:26.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c66ffd640 1 -- 172.21.15.16:0/2324343855 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4222879545 0 0) 0x7f0c50003520 con 0x7f0c6815bcd0 2024-04-24T23:50:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c66ffd640 1 -- 172.21.15.16:0/2324343855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c680d16c0 msgr2=0x7f0c681601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c66ffd640 1 --2- 172.21.15.16:0/2324343855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c680d16c0 0x7f0c681601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c66ffd640 1 -- 172.21.15.16:0/2324343855 >> v2:172.21.15.156:3300/0 conn(0x7f0c680705b0 msgr2=0x7f0c6806faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c66ffd640 1 --2- 172.21.15.16:0/2324343855 >> v2:172.21.15.156:3300/0 conn(0x7f0c680705b0 0x7f0c6806faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c66ffd640 1 -- 172.21.15.16:0/2324343855 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6806f010 con 0x7f0c6815bcd0 2024-04-24T23:50:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c66ffd640 1 -- 172.21.15.16:0/2324343855 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f0c5c001670 con 0x7f0c6815bcd0 2024-04-24T23:50:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c66ffd640 1 -- 172.21.15.16:0/2324343855 >> v1:172.21.15.16:6789/0 conn(0x7f0c6815bcd0 legacy=0x7f0c680d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c66ffd640 1 --2- 172.21.15.16:0/2324343855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c500039b0 0x7f0c50003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c66ffd640 1 --2- 172.21.15.16:0/2324343855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c680d16c0 0x7f0c50004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c66ffd640 1 --2- 172.21.15.16:0/2324343855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c50004d40 0x7f0c500051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c66ffd640 1 -- 172.21.15.16:0/2324343855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6806f660 con 0x7f0c500039b0 2024-04-24T23:50:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c66ffd640 1 -- 172.21.15.16:0/2324343855 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6806ed30 con 0x7f0c50004d40 2024-04-24T23:50:26.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c66ffd640 1 -- 172.21.15.16:0/2324343855 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0c5000ba40 con 0x7f0c680d16c0 2024-04-24T23:50:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c67fff640 1 --2- 172.21.15.16:0/2324343855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c500039b0 0x7f0c50003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c677fe640 1 --2- 172.21.15.16:0/2324343855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c680d16c0 0x7f0c50004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c6c8d8640 1 --2- 172.21.15.16:0/2324343855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c50004d40 0x7f0c500051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c67fff640 1 -- 172.21.15.16:0/2324343855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c680d16c0 msgr2=0x7f0c50004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c67fff640 1 --2- 172.21.15.16:0/2324343855 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c680d16c0 0x7f0c50004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c67fff640 1 -- 172.21.15.16:0/2324343855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c50004d40 msgr2=0x7f0c500051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c67fff640 1 --2- 172.21.15.16:0/2324343855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c50004d40 0x7f0c500051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.656+0000 7f0c67fff640 1 -- 172.21.15.16:0/2324343855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c50003520 con 0x7f0c500039b0 2024-04-24T23:50:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.657+0000 7f0c67fff640 1 --2- 172.21.15.16:0/2324343855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c500039b0 0x7f0c50003dd0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00c290 tx=0x7f0c50005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec0169cdf190c3c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.657+0000 7f0c66ffd640 1 -- 172.21.15.16:0/2324343855 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c007d50 con 0x7f0c500039b0 2024-04-24T23:50:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.657+0000 7f0c66ffd640 1 -- 172.21.15.16:0/2324343855 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c5c005af0 con 0x7f0c500039b0 2024-04-24T23:50:26.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.657+0000 7f0c66ffd640 1 -- 172.21.15.16:0/2324343855 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c007070 con 0x7f0c500039b0 2024-04-24T23:50:26.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.658+0000 7f0c6e33b640 1 -- 172.21.15.16:0/2324343855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c500039b0 msgr2=0x7f0c50003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:26.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.658+0000 7f0c6e33b640 1 --2- 172.21.15.16:0/2324343855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c500039b0 0x7f0c50003dd0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00c290 tx=0x7f0c50005830 comp rx=0 tx=0).stop 2024-04-24T23:50:26.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.658+0000 7f0c6c8d8640 1 -- 172.21.15.16:0/2324343855 reap_dead start 2024-04-24T23:50:26.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.658+0000 7f0c6e33b640 1 -- 172.21.15.16:0/2324343855 shutdown_connections 2024-04-24T23:50:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.658+0000 7f0c6e33b640 1 -- 172.21.15.16:0/2324343855 >> 172.21.15.16:0/2324343855 conn(0x7f0c6806b4d0 msgr2=0x7f0c680b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.658+0000 7f0c6e33b640 1 -- 172.21.15.16:0/2324343855 shutdown_connections 2024-04-24T23:50:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.658+0000 7f0c6e33b640 1 -- 172.21.15.16:0/2324343855 wait complete. 2024-04-24T23:50:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.659+0000 7f0c6e33b640 1 Processor -- start 2024-04-24T23:50:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.659+0000 7f0c6e33b640 1 -- start start 2024-04-24T23:50:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.660+0000 7f0c6e33b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c500039b0 0x7f0c6814a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.660+0000 7f0c6e33b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c680c7cf0 0x7f0c6815ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.660+0000 7f0c6e33b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c68155c40 0x7f0c681560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.660+0000 7f0c6e33b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0c680d09d0 con 0x7f0c500039b0 2024-04-24T23:50:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.660+0000 7f0c6e33b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0c6815c0b0 con 0x7f0c68155c40 2024-04-24T23:50:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.660+0000 7f0c6e33b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0c680d0b50 con 0x7f0c680c7cf0 2024-04-24T23:50:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.660+0000 7f0c67fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c500039b0 0x7f0c6814a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.660+0000 7f0c67fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c500039b0 0x7f0c6814a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38022/0 (socket says 172.21.15.16:38022) 2024-04-24T23:50:26.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.660+0000 7f0c67fff640 1 -- 172.21.15.16:0/770446600 learned_addr learned my addr 172.21.15.16:0/770446600 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:26.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.660+0000 7f0c6c8d8640 1 --2- 172.21.15.16:0/770446600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c68155c40 0x7f0c681560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:26.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.660+0000 7f0c677fe640 1 --2- 172.21.15.16:0/770446600 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c680c7cf0 0x7f0c6815ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:26.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.661+0000 7f0c67fff640 1 -- 172.21.15.16:0/770446600 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c680c7cf0 msgr2=0x7f0c6815ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:26.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.661+0000 7f0c67fff640 1 --2- 172.21.15.16:0/770446600 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c680c7cf0 0x7f0c6815ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:26.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.661+0000 7f0c67fff640 1 -- 172.21.15.16:0/770446600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c68155c40 msgr2=0x7f0c681560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:26.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.661+0000 7f0c67fff640 1 --2- 172.21.15.16:0/770446600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c68155c40 0x7f0c681560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:26.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.661+0000 7f0c67fff640 1 -- 172.21.15.16:0/770446600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c6815d620 con 0x7f0c500039b0 2024-04-24T23:50:26.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.661+0000 7f0c67fff640 1 --2- 172.21.15.16:0/770446600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c500039b0 0x7f0c6814a100 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00c3b0 tx=0x7f0c681609a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:26.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.661+0000 7f0c64ff9640 1 -- 172.21.15.16:0/770446600 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c042070 con 0x7f0c500039b0 2024-04-24T23:50:26.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.661+0000 7f0c64ff9640 1 -- 172.21.15.16:0/770446600 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c5c007480 con 0x7f0c500039b0 2024-04-24T23:50:26.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.661+0000 7f0c6e33b640 1 -- 172.21.15.16:0/770446600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c6815d900 con 0x7f0c500039b0 2024-04-24T23:50:26.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.661+0000 7f0c64ff9640 1 -- 172.21.15.16:0/770446600 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c5c001c80 con 0x7f0c500039b0 2024-04-24T23:50:26.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.661+0000 7f0c6e33b640 1 -- 172.21.15.16:0/770446600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c6815dec0 con 0x7f0c500039b0 2024-04-24T23:50:26.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.662+0000 7f0c6e33b640 1 -- 172.21.15.16:0/770446600 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c6806bea0 con 0x7f0c500039b0 2024-04-24T23:50:26.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.663+0000 7f0c64ff9640 1 -- 172.21.15.16:0/770446600 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0c5c045070 con 0x7f0c500039b0 2024-04-24T23:50:26.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.663+0000 7f0c64ff9640 1 --2- 172.21.15.16:0/770446600 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0c5406be30 0x7f0c5406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:26.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.663+0000 7f0c64ff9640 1 -- 172.21.15.16:0/770446600 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f0c5c0abaf0 con 0x7f0c500039b0 2024-04-24T23:50:26.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.663+0000 7f0c677fe640 1 --2- 172.21.15.16:0/770446600 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0c5406be30 0x7f0c5406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:26.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.664+0000 7f0c677fe640 1 --2- 172.21.15.16:0/770446600 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0c5406be30 0x7f0c5406e2f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0c680c80f0 tx=0x7f0c58009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:26.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.668+0000 7f0c64ff9640 1 -- 172.21.15.16:0/770446600 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0c5c005ac0 con 0x7f0c500039b0 2024-04-24T23:50:26.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.847+0000 7f0c6e33b640 1 -- 172.21.15.16:0/770446600 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0c6815e6c0 con 0x7f0c5406be30 2024-04-24T23:50:26.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.854+0000 7f0c64ff9640 1 -- 172.21.15.16:0/770446600 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251955 (secure 0 0 0) 0x7f0c6815e6c0 con 0x7f0c5406be30 2024-04-24T23:50:26.856 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:26.858 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-24T23:50:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.857+0000 7f0c6e33b640 1 -- 172.21.15.16:0/770446600 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0c5406be30 msgr2=0x7f0c5406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.857+0000 7f0c6e33b640 1 --2- 172.21.15.16:0/770446600 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0c5406be30 0x7f0c5406e2f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0c680c80f0 tx=0x7f0c58009190 comp rx=0 tx=0).stop 2024-04-24T23:50:26.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.857+0000 7f0c6e33b640 1 -- 172.21.15.16:0/770446600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c500039b0 msgr2=0x7f0c6814a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.857+0000 7f0c6e33b640 1 --2- 172.21.15.16:0/770446600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c500039b0 0x7f0c6814a100 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f0c5c00c3b0 tx=0x7f0c681609a0 comp rx=0 tx=0).stop 2024-04-24T23:50:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.857+0000 7f0c6e33b640 1 -- 172.21.15.16:0/770446600 shutdown_connections 2024-04-24T23:50:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.857+0000 7f0c6e33b640 1 --2- 172.21.15.16:0/770446600 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0c5406be30 0x7f0c5406e2f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.857+0000 7f0c6e33b640 1 --2- 172.21.15.16:0/770446600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0c68155c40 0x7f0c681560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.857+0000 7f0c6e33b640 1 --2- 172.21.15.16:0/770446600 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0c680c7cf0 0x7f0c6815ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.857+0000 7f0c6e33b640 1 --2- 172.21.15.16:0/770446600 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c500039b0 0x7f0c6814a100 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.857+0000 7f0c6e33b640 1 -- 172.21.15.16:0/770446600 >> 172.21.15.16:0/770446600 conn(0x7f0c6806b4d0 msgr2=0x7f0c680cd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.857+0000 7f0c6e33b640 1 -- 172.21.15.16:0/770446600 shutdown_connections 2024-04-24T23:50:26.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:26.857+0000 7f0c6e33b640 1 -- 172.21.15.16:0/770446600 wait complete. 2024-04-24T23:50:26.868 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-04-24T23:50:25.141138+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":110,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254184,"kb_used_data":4744,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720536,"statfs":{"total":773094113280,"available":772833828864,"internally_reserved":0,"allocated":4857856,"data_stored":2743212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.072844"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788401+0000","last_change":"2024-04-24T23:50:19.096331+0000","last_active":"2024-04-24T23:50:20.788401+0000","last_peered":"2024-04-24T23:50:20.788401+0000","last_clean":"2024-04-24T23:50:20.788401+0000","last_became_active":"2024-04-24T23:50:19.095755+0000","last_became_peered":"2024-04-24T23:50:19.095755+0000","last_unstale":"2024-04-24T23:50:20.788401+0000","last_undegraded":"2024-04-24T23:50:20.788401+0000","last_fullsized":"2024-04-24T23:50:20.788401+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:16:23.638767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788156+0000","last_change":"2024-04-24T23:50:19.086054+0000","last_active":"2024-04-24T23:50:20.788156+0000","last_peered":"2024-04-24T23:50:20.788156+0000","last_clean":"2024-04-24T23:50:20.788156+0000","last_became_active":"2024-04-24T23:50:19.085407+0000","last_became_peered":"2024-04-24T23:50:19.085407+0000","last_unstale":"2024-04-24T23:50:20.788156+0000","last_undegraded":"2024-04-24T23:50:20.788156+0000","last_fullsized":"2024-04-24T23:50:20.788156+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:44.423218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788242+0000","last_change":"2024-04-24T23:50:19.085231+0000","last_active":"2024-04-24T23:50:20.788242+0000","last_peered":"2024-04-24T23:50:20.788242+0000","last_clean":"2024-04-24T23:50:20.788242+0000","last_became_active":"2024-04-24T23:50:19.084968+0000","last_became_peered":"2024-04-24T23:50:19.084968+0000","last_unstale":"2024-04-24T23:50:20.788242+0000","last_undegraded":"2024-04-24T23:50:20.788242+0000","last_fullsized":"2024-04-24T23:50:20.788242+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:39:01.832638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788692+0000","last_change":"2024-04-24T23:50:19.090227+0000","last_active":"2024-04-24T23:50:20.788692+0000","last_peered":"2024-04-24T23:50:20.788692+0000","last_clean":"2024-04-24T23:50:20.788692+0000","last_became_active":"2024-04-24T23:50:19.089634+0000","last_became_peered":"2024-04-24T23:50:19.089634+0000","last_unstale":"2024-04-24T23:50:20.788692+0000","last_undegraded":"2024-04-24T23:50:20.788692+0000","last_fullsized":"2024-04-24T23:50:20.788692+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:10:02.733361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788678+0000","last_change":"2024-04-24T23:50:19.089138+0000","last_active":"2024-04-24T23:50:20.788678+0000","last_peered":"2024-04-24T23:50:20.788678+0000","last_clean":"2024-04-24T23:50:20.788678+0000","last_became_active":"2024-04-24T23:50:19.088868+0000","last_became_peered":"2024-04-24T23:50:19.088868+0000","last_unstale":"2024-04-24T23:50:20.788678+0000","last_undegraded":"2024-04-24T23:50:20.788678+0000","last_fullsized":"2024-04-24T23:50:20.788678+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:08:42.255450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.071816+0000","last_change":"2024-04-24T23:50:19.085098+0000","last_active":"2024-04-24T23:50:20.071816+0000","last_peered":"2024-04-24T23:50:20.071816+0000","last_clean":"2024-04-24T23:50:20.071816+0000","last_became_active":"2024-04-24T23:50:19.084806+0000","last_became_peered":"2024-04-24T23:50:19.084806+0000","last_unstale":"2024-04-24T23:50:20.071816+0000","last_undegraded":"2024-04-24T23:50:20.071816+0000","last_fullsized":"2024-04-24T23:50:20.071816+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:59.514575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.787948+0000","last_change":"2024-04-24T23:50:19.081837+0000","last_active":"2024-04-24T23:50:20.787948+0000","last_peered":"2024-04-24T23:50:20.787948+0000","last_clean":"2024-04-24T23:50:20.787948+0000","last_became_active":"2024-04-24T23:50:19.081526+0000","last_became_peered":"2024-04-24T23:50:19.081526+0000","last_unstale":"2024-04-24T23:50:20.787948+0000","last_undegraded":"2024-04-24T23:50:20.787948+0000","last_fullsized":"2024-04-24T23:50:20.787948+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:49:15.545291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.787850+0000","last_change":"2024-04-24T23:50:19.085713+0000","last_active":"2024-04-24T23:50:20.787850+0000","last_peered":"2024-04-24T23:50:20.787850+0000","last_clean":"2024-04-24T23:50:20.787850+0000","last_became_active":"2024-04-24T23:50:19.085385+0000","last_became_peered":"2024-04-24T23:50:19.085385+0000","last_unstale":"2024-04-24T23:50:20.787850+0000","last_undegraded":"2024-04-24T23:50:20.787850+0000","last_fullsized":"2024-04-24T23:50:20.787850+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:58.003442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.789200+0000","last_change":"2024-04-24T23:50:19.089120+0000","last_active":"2024-04-24T23:50:20.789200+0000","last_peered":"2024-04-24T23:50:20.789200+0000","last_clean":"2024-04-24T23:50:20.789200+0000","last_became_active":"2024-04-24T23:50:19.088862+0000","last_became_peered":"2024-04-24T23:50:19.088862+0000","last_unstale":"2024-04-24T23:50:20.789200+0000","last_undegraded":"2024-04-24T23:50:20.789200+0000","last_fullsized":"2024-04-24T23:50:20.789200+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:54:49.964613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788361+0000","last_change":"2024-04-24T23:50:19.079394+0000","last_active":"2024-04-24T23:50:20.788361+0000","last_peered":"2024-04-24T23:50:20.788361+0000","last_clean":"2024-04-24T23:50:20.788361+0000","last_became_active":"2024-04-24T23:50:19.079012+0000","last_became_peered":"2024-04-24T23:50:19.079012+0000","last_unstale":"2024-04-24T23:50:20.788361+0000","last_undegraded":"2024-04-24T23:50:20.788361+0000","last_fullsized":"2024-04-24T23:50:20.788361+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:16:23.638767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788469+0000","last_change":"2024-04-24T23:50:19.091815+0000","last_active":"2024-04-24T23:50:20.788469+0000","last_peered":"2024-04-24T23:50:20.788469+0000","last_clean":"2024-04-24T23:50:20.788469+0000","last_became_active":"2024-04-24T23:50:19.091486+0000","last_became_peered":"2024-04-24T23:50:19.091486+0000","last_unstale":"2024-04-24T23:50:20.788469+0000","last_undegraded":"2024-04-24T23:50:20.788469+0000","last_fullsized":"2024-04-24T23:50:20.788469+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:35:58.858220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788075+0000","last_change":"2024-04-24T23:50:19.086585+0000","last_active":"2024-04-24T23:50:20.788075+0000","last_peered":"2024-04-24T23:50:20.788075+0000","last_clean":"2024-04-24T23:50:20.788075+0000","last_became_active":"2024-04-24T23:50:19.086325+0000","last_became_peered":"2024-04-24T23:50:19.086325+0000","last_unstale":"2024-04-24T23:50:20.788075+0000","last_undegraded":"2024-04-24T23:50:20.788075+0000","last_fullsized":"2024-04-24T23:50:20.788075+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:03.073135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788110+0000","last_change":"2024-04-24T23:50:19.079371+0000","last_active":"2024-04-24T23:50:20.788110+0000","last_peered":"2024-04-24T23:50:20.788110+0000","last_clean":"2024-04-24T23:50:20.788110+0000","last_became_active":"2024-04-24T23:50:19.079080+0000","last_became_peered":"2024-04-24T23:50:19.079080+0000","last_unstale":"2024-04-24T23:50:20.788110+0000","last_undegraded":"2024-04-24T23:50:20.788110+0000","last_fullsized":"2024-04-24T23:50:20.788110+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:00.216226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075692+0000","last_change":"2024-04-24T23:50:19.086958+0000","last_active":"2024-04-24T23:50:20.075692+0000","last_peered":"2024-04-24T23:50:20.075692+0000","last_clean":"2024-04-24T23:50:20.075692+0000","last_became_active":"2024-04-24T23:50:19.086664+0000","last_became_peered":"2024-04-24T23:50:19.086664+0000","last_unstale":"2024-04-24T23:50:20.075692+0000","last_undegraded":"2024-04-24T23:50:20.075692+0000","last_fullsized":"2024-04-24T23:50:20.075692+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:03.073135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788040+0000","last_change":"2024-04-24T23:50:19.089302+0000","last_active":"2024-04-24T23:50:20.788040+0000","last_peered":"2024-04-24T23:50:20.788040+0000","last_clean":"2024-04-24T23:50:20.788040+0000","last_became_active":"2024-04-24T23:50:19.086389+0000","last_became_peered":"2024-04-24T23:50:19.086389+0000","last_unstale":"2024-04-24T23:50:20.788040+0000","last_undegraded":"2024-04-24T23:50:20.788040+0000","last_fullsized":"2024-04-24T23:50:20.788040+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:44.423218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074130+0000","last_change":"2024-04-24T23:50:19.083004+0000","last_active":"2024-04-24T23:50:20.074130+0000","last_peered":"2024-04-24T23:50:20.074130+0000","last_clean":"2024-04-24T23:50:20.074130+0000","last_became_active":"2024-04-24T23:50:19.082723+0000","last_became_peered":"2024-04-24T23:50:19.082723+0000","last_unstale":"2024-04-24T23:50:20.074130+0000","last_undegraded":"2024-04-24T23:50:20.074130+0000","last_fullsized":"2024-04-24T23:50:20.074130+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:30:46.624588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075281+0000","last_change":"2024-04-24T23:50:19.082525+0000","last_active":"2024-04-24T23:50:20.075281+0000","last_peered":"2024-04-24T23:50:20.075281+0000","last_clean":"2024-04-24T23:50:20.075281+0000","last_became_active":"2024-04-24T23:50:19.081187+0000","last_became_peered":"2024-04-24T23:50:19.081187+0000","last_unstale":"2024-04-24T23:50:20.075281+0000","last_undegraded":"2024-04-24T23:50:20.075281+0000","last_fullsized":"2024-04-24T23:50:20.075281+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:42:41.658151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073673+0000","last_change":"2024-04-24T23:50:19.089420+0000","last_active":"2024-04-24T23:50:20.073673+0000","last_peered":"2024-04-24T23:50:20.073673+0000","last_clean":"2024-04-24T23:50:20.073673+0000","last_became_active":"2024-04-24T23:50:19.089197+0000","last_became_peered":"2024-04-24T23:50:19.089197+0000","last_unstale":"2024-04-24T23:50:20.073673+0000","last_undegraded":"2024-04-24T23:50:20.073673+0000","last_fullsized":"2024-04-24T23:50:20.073673+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:50.774708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788464+0000","last_change":"2024-04-24T23:50:19.086234+0000","last_active":"2024-04-24T23:50:20.788464+0000","last_peered":"2024-04-24T23:50:20.788464+0000","last_clean":"2024-04-24T23:50:20.788464+0000","last_became_active":"2024-04-24T23:50:19.085987+0000","last_became_peered":"2024-04-24T23:50:19.085987+0000","last_unstale":"2024-04-24T23:50:20.788464+0000","last_undegraded":"2024-04-24T23:50:20.788464+0000","last_fullsized":"2024-04-24T23:50:20.788464+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:51:04.482202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073441+0000","last_change":"2024-04-24T23:50:19.090305+0000","last_active":"2024-04-24T23:50:20.073441+0000","last_peered":"2024-04-24T23:50:20.073441+0000","last_clean":"2024-04-24T23:50:20.073441+0000","last_became_active":"2024-04-24T23:50:19.090023+0000","last_became_peered":"2024-04-24T23:50:19.090023+0000","last_unstale":"2024-04-24T23:50:20.073441+0000","last_undegraded":"2024-04-24T23:50:20.073441+0000","last_fullsized":"2024-04-24T23:50:20.073441+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:56.437091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788183+0000","last_change":"2024-04-24T23:50:19.086653+0000","last_active":"2024-04-24T23:50:20.788183+0000","last_peered":"2024-04-24T23:50:20.788183+0000","last_clean":"2024-04-24T23:50:20.788183+0000","last_became_active":"2024-04-24T23:50:19.086376+0000","last_became_peered":"2024-04-24T23:50:19.086376+0000","last_unstale":"2024-04-24T23:50:20.788183+0000","last_undegraded":"2024-04-24T23:50:20.788183+0000","last_fullsized":"2024-04-24T23:50:20.788183+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:59.007356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.076072+0000","last_change":"2024-04-24T23:50:19.093829+0000","last_active":"2024-04-24T23:50:20.076072+0000","last_peered":"2024-04-24T23:50:20.076072+0000","last_clean":"2024-04-24T23:50:20.076072+0000","last_became_active":"2024-04-24T23:50:19.092122+0000","last_became_peered":"2024-04-24T23:50:19.092122+0000","last_unstale":"2024-04-24T23:50:20.076072+0000","last_undegraded":"2024-04-24T23:50:20.076072+0000","last_fullsized":"2024-04-24T23:50:20.076072+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:59.514575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074752+0000","last_change":"2024-04-24T23:50:19.079759+0000","last_active":"2024-04-24T23:50:20.074752+0000","last_peered":"2024-04-24T23:50:20.074752+0000","last_clean":"2024-04-24T23:50:20.074752+0000","last_became_active":"2024-04-24T23:50:19.079496+0000","last_became_peered":"2024-04-24T23:50:19.079496+0000","last_unstale":"2024-04-24T23:50:20.074752+0000","last_undegraded":"2024-04-24T23:50:20.074752+0000","last_fullsized":"2024-04-24T23:50:20.074752+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:19:09.036785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788031+0000","last_change":"2024-04-24T23:50:19.077391+0000","last_active":"2024-04-24T23:50:20.788031+0000","last_peered":"2024-04-24T23:50:20.788031+0000","last_clean":"2024-04-24T23:50:20.788031+0000","last_became_active":"2024-04-24T23:50:19.077099+0000","last_became_peered":"2024-04-24T23:50:19.077099+0000","last_unstale":"2024-04-24T23:50:20.788031+0000","last_undegraded":"2024-04-24T23:50:20.788031+0000","last_fullsized":"2024-04-24T23:50:20.788031+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:55:28.732222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073952+0000","last_change":"2024-04-24T23:50:19.089032+0000","last_active":"2024-04-24T23:50:20.073952+0000","last_peered":"2024-04-24T23:50:20.073952+0000","last_clean":"2024-04-24T23:50:20.073952+0000","last_became_active":"2024-04-24T23:50:19.088739+0000","last_became_peered":"2024-04-24T23:50:19.088739+0000","last_unstale":"2024-04-24T23:50:20.073952+0000","last_undegraded":"2024-04-24T23:50:20.073952+0000","last_fullsized":"2024-04-24T23:50:20.073952+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:39:01.832638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.072104+0000","last_change":"2024-04-24T23:50:19.083395+0000","last_active":"2024-04-24T23:50:20.072104+0000","last_peered":"2024-04-24T23:50:20.072104+0000","last_clean":"2024-04-24T23:50:20.072104+0000","last_became_active":"2024-04-24T23:50:19.083139+0000","last_became_peered":"2024-04-24T23:50:19.083139+0000","last_unstale":"2024-04-24T23:50:20.072104+0000","last_undegraded":"2024-04-24T23:50:20.072104+0000","last_fullsized":"2024-04-24T23:50:20.072104+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:54:49.964613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073804+0000","last_change":"2024-04-24T23:50:19.081999+0000","last_active":"2024-04-24T23:50:20.073804+0000","last_peered":"2024-04-24T23:50:20.073804+0000","last_clean":"2024-04-24T23:50:20.073804+0000","last_became_active":"2024-04-24T23:50:19.081681+0000","last_became_peered":"2024-04-24T23:50:19.081681+0000","last_unstale":"2024-04-24T23:50:20.073804+0000","last_undegraded":"2024-04-24T23:50:20.073804+0000","last_fullsized":"2024-04-24T23:50:20.073804+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:03.073135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788421+0000","last_change":"2024-04-24T23:50:19.091112+0000","last_active":"2024-04-24T23:50:20.788421+0000","last_peered":"2024-04-24T23:50:20.788421+0000","last_clean":"2024-04-24T23:50:20.788421+0000","last_became_active":"2024-04-24T23:50:19.090810+0000","last_became_peered":"2024-04-24T23:50:19.090810+0000","last_unstale":"2024-04-24T23:50:20.788421+0000","last_undegraded":"2024-04-24T23:50:20.788421+0000","last_fullsized":"2024-04-24T23:50:20.788421+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:05:26.687147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074013+0000","last_change":"2024-04-24T23:50:19.093267+0000","last_active":"2024-04-24T23:50:20.074013+0000","last_peered":"2024-04-24T23:50:20.074013+0000","last_clean":"2024-04-24T23:50:20.074013+0000","last_became_active":"2024-04-24T23:50:19.092999+0000","last_became_peered":"2024-04-24T23:50:19.092999+0000","last_unstale":"2024-04-24T23:50:20.074013+0000","last_undegraded":"2024-04-24T23:50:20.074013+0000","last_fullsized":"2024-04-24T23:50:20.074013+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:39.114823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074577+0000","last_change":"2024-04-24T23:50:19.083414+0000","last_active":"2024-04-24T23:50:20.074577+0000","last_peered":"2024-04-24T23:50:20.074577+0000","last_clean":"2024-04-24T23:50:20.074577+0000","last_became_active":"2024-04-24T23:50:19.083007+0000","last_became_peered":"2024-04-24T23:50:19.083007+0000","last_unstale":"2024-04-24T23:50:20.074577+0000","last_undegraded":"2024-04-24T23:50:20.074577+0000","last_fullsized":"2024-04-24T23:50:20.074577+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:59.514575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074759+0000","last_change":"2024-04-24T23:50:19.079184+0000","last_active":"2024-04-24T23:50:20.074759+0000","last_peered":"2024-04-24T23:50:20.074759+0000","last_clean":"2024-04-24T23:50:20.074759+0000","last_became_active":"2024-04-24T23:50:19.078943+0000","last_became_peered":"2024-04-24T23:50:19.078943+0000","last_unstale":"2024-04-24T23:50:20.074759+0000","last_undegraded":"2024-04-24T23:50:20.074759+0000","last_fullsized":"2024-04-24T23:50:20.074759+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:44:04.875514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.072053+0000","last_change":"2024-04-24T23:50:19.084875+0000","last_active":"2024-04-24T23:50:20.072053+0000","last_peered":"2024-04-24T23:50:20.072053+0000","last_clean":"2024-04-24T23:50:20.072053+0000","last_became_active":"2024-04-24T23:50:19.084583+0000","last_became_peered":"2024-04-24T23:50:19.084583+0000","last_unstale":"2024-04-24T23:50:20.072053+0000","last_undegraded":"2024-04-24T23:50:20.072053+0000","last_fullsized":"2024-04-24T23:50:20.072053+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:53.415628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075059+0000","last_change":"2024-04-24T23:50:19.077387+0000","last_active":"2024-04-24T23:50:20.075059+0000","last_peered":"2024-04-24T23:50:20.075059+0000","last_clean":"2024-04-24T23:50:20.075059+0000","last_became_active":"2024-04-24T23:50:19.077072+0000","last_became_peered":"2024-04-24T23:50:19.077072+0000","last_unstale":"2024-04-24T23:50:20.075059+0000","last_undegraded":"2024-04-24T23:50:20.075059+0000","last_fullsized":"2024-04-24T23:50:20.075059+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:17.465127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"38'4","reported_seq":18,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073963+0000","last_change":"2024-04-24T23:50:18.085951+0000","last_active":"2024-04-24T23:50:20.073963+0000","last_peered":"2024-04-24T23:50:20.073963+0000","last_clean":"2024-04-24T23:50:20.073963+0000","last_became_active":"2024-04-24T23:50:18.085687+0000","last_became_peered":"2024-04-24T23:50:18.085687+0000","last_unstale":"2024-04-24T23:50:20.073963+0000","last_undegraded":"2024-04-24T23:50:20.073963+0000","last_fullsized":"2024-04-24T23:50:20.073963+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:36:44.740515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.072032+0000","last_change":"2024-04-24T23:50:19.084128+0000","last_active":"2024-04-24T23:50:20.072032+0000","last_peered":"2024-04-24T23:50:20.072032+0000","last_clean":"2024-04-24T23:50:20.072032+0000","last_became_active":"2024-04-24T23:50:19.083841+0000","last_became_peered":"2024-04-24T23:50:19.083841+0000","last_unstale":"2024-04-24T23:50:20.072032+0000","last_undegraded":"2024-04-24T23:50:20.072032+0000","last_fullsized":"2024-04-24T23:50:20.072032+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:20.973639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788569+0000","last_change":"2024-04-24T23:50:18.086468+0000","last_active":"2024-04-24T23:50:20.788569+0000","last_peered":"2024-04-24T23:50:20.788569+0000","last_clean":"2024-04-24T23:50:20.788569+0000","last_became_active":"2024-04-24T23:50:18.086193+0000","last_became_peered":"2024-04-24T23:50:18.086193+0000","last_unstale":"2024-04-24T23:50:20.788569+0000","last_undegraded":"2024-04-24T23:50:20.788569+0000","last_fullsized":"2024-04-24T23:50:20.788569+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:36:44.740515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.076139+0000","last_change":"2024-04-24T23:50:19.088008+0000","last_active":"2024-04-24T23:50:20.076139+0000","last_peered":"2024-04-24T23:50:20.076139+0000","last_clean":"2024-04-24T23:50:20.076139+0000","last_became_active":"2024-04-24T23:50:19.087619+0000","last_became_peered":"2024-04-24T23:50:19.087619+0000","last_unstale":"2024-04-24T23:50:20.076139+0000","last_undegraded":"2024-04-24T23:50:20.076139+0000","last_fullsized":"2024-04-24T23:50:20.076139+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:10.966605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.072081+0000","last_change":"2024-04-24T23:50:18.084764+0000","last_active":"2024-04-24T23:50:20.072081+0000","last_peered":"2024-04-24T23:50:20.072081+0000","last_clean":"2024-04-24T23:50:20.072081+0000","last_became_active":"2024-04-24T23:50:18.084197+0000","last_became_peered":"2024-04-24T23:50:18.084197+0000","last_unstale":"2024-04-24T23:50:20.072081+0000","last_undegraded":"2024-04-24T23:50:20.072081+0000","last_fullsized":"2024-04-24T23:50:20.072081+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:23:27.985620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788257+0000","last_change":"2024-04-24T23:50:19.087571+0000","last_active":"2024-04-24T23:50:20.788257+0000","last_peered":"2024-04-24T23:50:20.788257+0000","last_clean":"2024-04-24T23:50:20.788257+0000","last_became_active":"2024-04-24T23:50:19.087171+0000","last_became_peered":"2024-04-24T23:50:19.087171+0000","last_unstale":"2024-04-24T23:50:20.788257+0000","last_undegraded":"2024-04-24T23:50:20.788257+0000","last_fullsized":"2024-04-24T23:50:20.788257+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:09:00.216226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075985+0000","last_change":"2024-04-24T23:50:18.088963+0000","last_active":"2024-04-24T23:50:20.075985+0000","last_peered":"2024-04-24T23:50:20.075985+0000","last_clean":"2024-04-24T23:50:20.075985+0000","last_became_active":"2024-04-24T23:50:18.088648+0000","last_became_peered":"2024-04-24T23:50:18.088648+0000","last_unstale":"2024-04-24T23:50:20.075985+0000","last_undegraded":"2024-04-24T23:50:20.075985+0000","last_fullsized":"2024-04-24T23:50:20.075985+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:19:42.318258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788657+0000","last_change":"2024-04-24T23:50:19.083448+0000","last_active":"2024-04-24T23:50:20.788657+0000","last_peered":"2024-04-24T23:50:20.788657+0000","last_clean":"2024-04-24T23:50:20.788657+0000","last_became_active":"2024-04-24T23:50:19.083197+0000","last_became_peered":"2024-04-24T23:50:19.083197+0000","last_unstale":"2024-04-24T23:50:20.788657+0000","last_undegraded":"2024-04-24T23:50:20.788657+0000","last_fullsized":"2024-04-24T23:50:20.788657+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:08:59.514575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788199+0000","last_change":"2024-04-24T23:50:18.088565+0000","last_active":"2024-04-24T23:50:20.788199+0000","last_peered":"2024-04-24T23:50:20.788199+0000","last_clean":"2024-04-24T23:50:20.788199+0000","last_became_active":"2024-04-24T23:50:18.087754+0000","last_became_peered":"2024-04-24T23:50:18.087754+0000","last_unstale":"2024-04-24T23:50:20.788199+0000","last_undegraded":"2024-04-24T23:50:20.788199+0000","last_fullsized":"2024-04-24T23:50:20.788199+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:49.773026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788207+0000","last_change":"2024-04-24T23:50:19.089083+0000","last_active":"2024-04-24T23:50:20.788207+0000","last_peered":"2024-04-24T23:50:20.788207+0000","last_clean":"2024-04-24T23:50:20.788207+0000","last_became_active":"2024-04-24T23:50:19.086994+0000","last_became_peered":"2024-04-24T23:50:19.086994+0000","last_unstale":"2024-04-24T23:50:20.788207+0000","last_undegraded":"2024-04-24T23:50:20.788207+0000","last_fullsized":"2024-04-24T23:50:20.788207+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:53.860893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073105+0000","last_change":"2024-04-24T23:50:18.084696+0000","last_active":"2024-04-24T23:50:20.073105+0000","last_peered":"2024-04-24T23:50:20.073105+0000","last_clean":"2024-04-24T23:50:20.073105+0000","last_became_active":"2024-04-24T23:50:18.084421+0000","last_became_peered":"2024-04-24T23:50:18.084421+0000","last_unstale":"2024-04-24T23:50:20.073105+0000","last_undegraded":"2024-04-24T23:50:20.073105+0000","last_fullsized":"2024-04-24T23:50:20.073105+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:09:52.280197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075868+0000","last_change":"2024-04-24T23:50:19.092620+0000","last_active":"2024-04-24T23:50:20.075868+0000","last_peered":"2024-04-24T23:50:20.075868+0000","last_clean":"2024-04-24T23:50:20.075868+0000","last_became_active":"2024-04-24T23:50:19.092166+0000","last_became_peered":"2024-04-24T23:50:19.092166+0000","last_unstale":"2024-04-24T23:50:20.075868+0000","last_undegraded":"2024-04-24T23:50:20.075868+0000","last_fullsized":"2024-04-24T23:50:20.075868+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:39.114823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.071743+0000","last_change":"2024-04-24T23:50:18.088633+0000","last_active":"2024-04-24T23:50:20.071743+0000","last_peered":"2024-04-24T23:50:20.071743+0000","last_clean":"2024-04-24T23:50:20.071743+0000","last_became_active":"2024-04-24T23:50:18.088327+0000","last_became_peered":"2024-04-24T23:50:18.088327+0000","last_unstale":"2024-04-24T23:50:20.071743+0000","last_undegraded":"2024-04-24T23:50:20.071743+0000","last_fullsized":"2024-04-24T23:50:20.071743+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:09:52.280197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788141+0000","last_change":"2024-04-24T23:50:19.096980+0000","last_active":"2024-04-24T23:50:20.788141+0000","last_peered":"2024-04-24T23:50:20.788141+0000","last_clean":"2024-04-24T23:50:20.788141+0000","last_became_active":"2024-04-24T23:50:19.096697+0000","last_became_peered":"2024-04-24T23:50:19.096697+0000","last_unstale":"2024-04-24T23:50:20.788141+0000","last_undegraded":"2024-04-24T23:50:20.788141+0000","last_fullsized":"2024-04-24T23:50:20.788141+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:49:15.545291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075551+0000","last_change":"2024-04-24T23:50:18.084321+0000","last_active":"2024-04-24T23:50:20.075551+0000","last_peered":"2024-04-24T23:50:20.075551+0000","last_clean":"2024-04-24T23:50:20.075551+0000","last_became_active":"2024-04-24T23:50:18.084049+0000","last_became_peered":"2024-04-24T23:50:18.084049+0000","last_unstale":"2024-04-24T23:50:20.075551+0000","last_undegraded":"2024-04-24T23:50:20.075551+0000","last_fullsized":"2024-04-24T23:50:20.075551+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:40.483306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788855+0000","last_change":"2024-04-24T23:50:19.090430+0000","last_active":"2024-04-24T23:50:20.788855+0000","last_peered":"2024-04-24T23:50:20.788855+0000","last_clean":"2024-04-24T23:50:20.788855+0000","last_became_active":"2024-04-24T23:50:19.089972+0000","last_became_peered":"2024-04-24T23:50:19.089972+0000","last_unstale":"2024-04-24T23:50:20.788855+0000","last_undegraded":"2024-04-24T23:50:20.788855+0000","last_fullsized":"2024-04-24T23:50:20.788855+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:26:35.411631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788447+0000","last_change":"2024-04-24T23:50:18.090464+0000","last_active":"2024-04-24T23:50:20.788447+0000","last_peered":"2024-04-24T23:50:20.788447+0000","last_clean":"2024-04-24T23:50:20.788447+0000","last_became_active":"2024-04-24T23:50:18.090177+0000","last_became_peered":"2024-04-24T23:50:18.090177+0000","last_unstale":"2024-04-24T23:50:20.788447+0000","last_undegraded":"2024-04-24T23:50:20.788447+0000","last_fullsized":"2024-04-24T23:50:20.788447+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:02.071453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075163+0000","last_change":"2024-04-24T23:50:19.084335+0000","last_active":"2024-04-24T23:50:20.075163+0000","last_peered":"2024-04-24T23:50:20.075163+0000","last_clean":"2024-04-24T23:50:20.075163+0000","last_became_active":"2024-04-24T23:50:19.084050+0000","last_became_peered":"2024-04-24T23:50:19.084050+0000","last_unstale":"2024-04-24T23:50:20.075163+0000","last_undegraded":"2024-04-24T23:50:20.075163+0000","last_fullsized":"2024-04-24T23:50:20.075163+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:49:15.545291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788379+0000","last_change":"2024-04-24T23:50:18.086806+0000","last_active":"2024-04-24T23:50:20.788379+0000","last_peered":"2024-04-24T23:50:20.788379+0000","last_clean":"2024-04-24T23:50:20.788379+0000","last_became_active":"2024-04-24T23:50:18.086489+0000","last_became_peered":"2024-04-24T23:50:18.086489+0000","last_unstale":"2024-04-24T23:50:20.788379+0000","last_undegraded":"2024-04-24T23:50:20.788379+0000","last_fullsized":"2024-04-24T23:50:20.788379+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:09:02.071453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788513+0000","last_change":"2024-04-24T23:50:19.091664+0000","last_active":"2024-04-24T23:50:20.788513+0000","last_peered":"2024-04-24T23:50:20.788513+0000","last_clean":"2024-04-24T23:50:20.788513+0000","last_became_active":"2024-04-24T23:50:19.090350+0000","last_became_peered":"2024-04-24T23:50:19.090350+0000","last_unstale":"2024-04-24T23:50:20.788513+0000","last_undegraded":"2024-04-24T23:50:20.788513+0000","last_fullsized":"2024-04-24T23:50:20.788513+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:39.114823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"38'3","reported_seq":17,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788516+0000","last_change":"2024-04-24T23:50:18.081470+0000","last_active":"2024-04-24T23:50:20.788516+0000","last_peered":"2024-04-24T23:50:20.788516+0000","last_clean":"2024-04-24T23:50:20.788516+0000","last_became_active":"2024-04-24T23:50:18.081021+0000","last_became_peered":"2024-04-24T23:50:18.081021+0000","last_unstale":"2024-04-24T23:50:20.788516+0000","last_undegraded":"2024-04-24T23:50:20.788516+0000","last_fullsized":"2024-04-24T23:50:20.788516+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:38.113141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.076180+0000","last_change":"2024-04-24T23:50:19.089223+0000","last_active":"2024-04-24T23:50:20.076180+0000","last_peered":"2024-04-24T23:50:20.076180+0000","last_clean":"2024-04-24T23:50:20.076180+0000","last_became_active":"2024-04-24T23:50:19.088972+0000","last_became_peered":"2024-04-24T23:50:19.088972+0000","last_unstale":"2024-04-24T23:50:20.076180+0000","last_undegraded":"2024-04-24T23:50:20.076180+0000","last_fullsized":"2024-04-24T23:50:20.076180+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:53.415628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"38'6","reported_seq":20,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788830+0000","last_change":"2024-04-24T23:50:18.086865+0000","last_active":"2024-04-24T23:50:20.788830+0000","last_peered":"2024-04-24T23:50:20.788830+0000","last_clean":"2024-04-24T23:50:20.788830+0000","last_became_active":"2024-04-24T23:50:18.086448+0000","last_became_peered":"2024-04-24T23:50:18.086448+0000","last_unstale":"2024-04-24T23:50:20.788830+0000","last_undegraded":"2024-04-24T23:50:20.788830+0000","last_fullsized":"2024-04-24T23:50:20.788830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:46:49.773026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788820+0000","last_change":"2024-04-24T23:50:19.088463+0000","last_active":"2024-04-24T23:50:20.788820+0000","last_peered":"2024-04-24T23:50:20.788820+0000","last_clean":"2024-04-24T23:50:20.788820+0000","last_became_active":"2024-04-24T23:50:19.088206+0000","last_became_peered":"2024-04-24T23:50:19.088206+0000","last_unstale":"2024-04-24T23:50:20.788820+0000","last_undegraded":"2024-04-24T23:50:20.788820+0000","last_fullsized":"2024-04-24T23:50:20.788820+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:10.966605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788258+0000","last_change":"2024-04-24T23:50:18.087815+0000","last_active":"2024-04-24T23:50:20.788258+0000","last_peered":"2024-04-24T23:50:20.788258+0000","last_clean":"2024-04-24T23:50:20.788258+0000","last_became_active":"2024-04-24T23:50:18.087511+0000","last_became_peered":"2024-04-24T23:50:18.087511+0000","last_unstale":"2024-04-24T23:50:20.788258+0000","last_undegraded":"2024-04-24T23:50:20.788258+0000","last_fullsized":"2024-04-24T23:50:20.788258+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:55.435409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.787972+0000","last_change":"2024-04-24T23:50:19.090999+0000","last_active":"2024-04-24T23:50:20.787972+0000","last_peered":"2024-04-24T23:50:20.787972+0000","last_clean":"2024-04-24T23:50:20.787972+0000","last_became_active":"2024-04-24T23:50:19.090687+0000","last_became_peered":"2024-04-24T23:50:19.090687+0000","last_unstale":"2024-04-24T23:50:20.787972+0000","last_undegraded":"2024-04-24T23:50:20.787972+0000","last_fullsized":"2024-04-24T23:50:20.787972+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T05:24:17.465127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.071872+0000","last_change":"2024-04-24T23:50:18.087633+0000","last_active":"2024-04-24T23:50:20.071872+0000","last_peered":"2024-04-24T23:50:20.071872+0000","last_clean":"2024-04-24T23:50:20.071872+0000","last_became_active":"2024-04-24T23:50:18.087254+0000","last_became_peered":"2024-04-24T23:50:18.087254+0000","last_unstale":"2024-04-24T23:50:20.071872+0000","last_undegraded":"2024-04-24T23:50:20.071872+0000","last_fullsized":"2024-04-24T23:50:20.071872+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:55:08.953626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073039+0000","last_change":"2024-04-24T23:50:19.078886+0000","last_active":"2024-04-24T23:50:20.073039+0000","last_peered":"2024-04-24T23:50:20.073039+0000","last_clean":"2024-04-24T23:50:20.073039+0000","last_became_active":"2024-04-24T23:50:19.078610+0000","last_became_peered":"2024-04-24T23:50:19.078610+0000","last_unstale":"2024-04-24T23:50:20.073039+0000","last_undegraded":"2024-04-24T23:50:20.073039+0000","last_fullsized":"2024-04-24T23:50:20.073039+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T23:54:49.964613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.073370+0000","last_change":"2024-04-24T23:50:18.084874+0000","last_active":"2024-04-24T23:50:20.073370+0000","last_peered":"2024-04-24T23:50:20.073370+0000","last_clean":"2024-04-24T23:50:20.073370+0000","last_became_active":"2024-04-24T23:50:18.083917+0000","last_became_peered":"2024-04-24T23:50:18.083917+0000","last_unstale":"2024-04-24T23:50:20.073370+0000","last_undegraded":"2024-04-24T23:50:20.073370+0000","last_fullsized":"2024-04-24T23:50:20.073370+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:40.483306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788197+0000","last_change":"2024-04-24T23:50:19.091744+0000","last_active":"2024-04-24T23:50:20.788197+0000","last_peered":"2024-04-24T23:50:20.788197+0000","last_clean":"2024-04-24T23:50:20.788197+0000","last_became_active":"2024-04-24T23:50:19.091084+0000","last_became_peered":"2024-04-24T23:50:19.091084+0000","last_unstale":"2024-04-24T23:50:20.788197+0000","last_undegraded":"2024-04-24T23:50:20.788197+0000","last_fullsized":"2024-04-24T23:50:20.788197+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:42:41.658151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"38'8","reported_seq":22,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074196+0000","last_change":"2024-04-24T23:50:18.085038+0000","last_active":"2024-04-24T23:50:20.074196+0000","last_peered":"2024-04-24T23:50:20.074196+0000","last_clean":"2024-04-24T23:50:20.074196+0000","last_became_active":"2024-04-24T23:50:18.084787+0000","last_became_peered":"2024-04-24T23:50:18.084787+0000","last_unstale":"2024-04-24T23:50:20.074196+0000","last_undegraded":"2024-04-24T23:50:20.074196+0000","last_fullsized":"2024-04-24T23:50:20.074196+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:30.330350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788648+0000","last_change":"2024-04-24T23:50:19.086313+0000","last_active":"2024-04-24T23:50:20.788648+0000","last_peered":"2024-04-24T23:50:20.788648+0000","last_clean":"2024-04-24T23:50:20.788648+0000","last_became_active":"2024-04-24T23:50:19.085565+0000","last_became_peered":"2024-04-24T23:50:19.085565+0000","last_unstale":"2024-04-24T23:50:20.788648+0000","last_undegraded":"2024-04-24T23:50:20.788648+0000","last_fullsized":"2024-04-24T23:50:20.788648+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:19.118407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788318+0000","last_change":"2024-04-24T23:50:18.081636+0000","last_active":"2024-04-24T23:50:20.788318+0000","last_peered":"2024-04-24T23:50:20.788318+0000","last_clean":"2024-04-24T23:50:20.788318+0000","last_became_active":"2024-04-24T23:50:18.081311+0000","last_became_peered":"2024-04-24T23:50:18.081311+0000","last_unstale":"2024-04-24T23:50:20.788318+0000","last_undegraded":"2024-04-24T23:50:20.788318+0000","last_fullsized":"2024-04-24T23:50:20.788318+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:12:55.435409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788468+0000","last_change":"2024-04-24T23:50:19.087011+0000","last_active":"2024-04-24T23:50:20.788468+0000","last_peered":"2024-04-24T23:50:20.788468+0000","last_clean":"2024-04-24T23:50:20.788468+0000","last_became_active":"2024-04-24T23:50:19.086515+0000","last_became_peered":"2024-04-24T23:50:19.086515+0000","last_unstale":"2024-04-24T23:50:20.788468+0000","last_undegraded":"2024-04-24T23:50:20.788468+0000","last_fullsized":"2024-04-24T23:50:20.788468+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:39:32.387993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.076195+0000","last_change":"2024-04-24T23:50:18.086714+0000","last_active":"2024-04-24T23:50:20.076195+0000","last_peered":"2024-04-24T23:50:20.076195+0000","last_clean":"2024-04-24T23:50:20.076195+0000","last_became_active":"2024-04-24T23:50:18.086282+0000","last_became_peered":"2024-04-24T23:50:18.086282+0000","last_unstale":"2024-04-24T23:50:20.076195+0000","last_undegraded":"2024-04-24T23:50:20.076195+0000","last_fullsized":"2024-04-24T23:50:20.076195+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:36:44.740515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.076131+0000","last_change":"2024-04-24T23:50:19.095372+0000","last_active":"2024-04-24T23:50:20.076131+0000","last_peered":"2024-04-24T23:50:20.076131+0000","last_clean":"2024-04-24T23:50:20.076131+0000","last_became_active":"2024-04-24T23:50:19.095122+0000","last_became_peered":"2024-04-24T23:50:19.095122+0000","last_unstale":"2024-04-24T23:50:20.076131+0000","last_undegraded":"2024-04-24T23:50:20.076131+0000","last_fullsized":"2024-04-24T23:50:20.076131+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:20.973639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"38'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788717+0000","last_change":"2024-04-24T23:50:18.076361+0000","last_active":"2024-04-24T23:50:20.788717+0000","last_peered":"2024-04-24T23:50:20.788717+0000","last_clean":"2024-04-24T23:50:20.788717+0000","last_became_active":"2024-04-24T23:50:18.076076+0000","last_became_peered":"2024-04-24T23:50:18.076076+0000","last_unstale":"2024-04-24T23:50:20.788717+0000","last_undegraded":"2024-04-24T23:50:20.788717+0000","last_fullsized":"2024-04-24T23:50:20.788717+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:03:10.961797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788847+0000","last_change":"2024-04-24T23:50:19.088399+0000","last_active":"2024-04-24T23:50:20.788847+0000","last_peered":"2024-04-24T23:50:20.788847+0000","last_clean":"2024-04-24T23:50:20.788847+0000","last_became_active":"2024-04-24T23:50:19.088106+0000","last_became_peered":"2024-04-24T23:50:19.088106+0000","last_unstale":"2024-04-24T23:50:20.788847+0000","last_undegraded":"2024-04-24T23:50:20.788847+0000","last_fullsized":"2024-04-24T23:50:20.788847+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:27:27.500102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.071862+0000","last_change":"2024-04-24T23:50:18.082589+0000","last_active":"2024-04-24T23:50:20.071862+0000","last_peered":"2024-04-24T23:50:20.071862+0000","last_clean":"2024-04-24T23:50:20.071862+0000","last_became_active":"2024-04-24T23:50:18.082312+0000","last_became_peered":"2024-04-24T23:50:18.082312+0000","last_unstale":"2024-04-24T23:50:20.071862+0000","last_undegraded":"2024-04-24T23:50:20.071862+0000","last_fullsized":"2024-04-24T23:50:20.071862+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:39:30.330350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074347+0000","last_change":"2024-04-24T23:50:19.079370+0000","last_active":"2024-04-24T23:50:20.074347+0000","last_peered":"2024-04-24T23:50:20.074347+0000","last_clean":"2024-04-24T23:50:20.074347+0000","last_became_active":"2024-04-24T23:50:19.079115+0000","last_became_peered":"2024-04-24T23:50:19.079115+0000","last_unstale":"2024-04-24T23:50:20.074347+0000","last_undegraded":"2024-04-24T23:50:20.074347+0000","last_fullsized":"2024-04-24T23:50:20.074347+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:12:12.871447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788464+0000","last_change":"2024-04-24T23:50:18.083687+0000","last_active":"2024-04-24T23:50:20.788464+0000","last_peered":"2024-04-24T23:50:20.788464+0000","last_clean":"2024-04-24T23:50:20.788464+0000","last_became_active":"2024-04-24T23:50:18.083380+0000","last_became_peered":"2024-04-24T23:50:18.083380+0000","last_unstale":"2024-04-24T23:50:20.788464+0000","last_undegraded":"2024-04-24T23:50:20.788464+0000","last_fullsized":"2024-04-24T23:50:20.788464+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:01:56.058772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788866+0000","last_change":"2024-04-24T23:50:19.089951+0000","last_active":"2024-04-24T23:50:20.788866+0000","last_peered":"2024-04-24T23:50:20.788866+0000","last_clean":"2024-04-24T23:50:20.788866+0000","last_became_active":"2024-04-24T23:50:19.089663+0000","last_became_peered":"2024-04-24T23:50:19.089663+0000","last_unstale":"2024-04-24T23:50:20.788866+0000","last_undegraded":"2024-04-24T23:50:20.788866+0000","last_fullsized":"2024-04-24T23:50:20.788866+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:48:20.973639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788341+0000","last_change":"2024-04-24T23:50:18.084068+0000","last_active":"2024-04-24T23:50:20.788341+0000","last_peered":"2024-04-24T23:50:20.788341+0000","last_clean":"2024-04-24T23:50:20.788341+0000","last_became_active":"2024-04-24T23:50:18.083571+0000","last_became_peered":"2024-04-24T23:50:18.083571+0000","last_unstale":"2024-04-24T23:50:20.788341+0000","last_undegraded":"2024-04-24T23:50:20.788341+0000","last_fullsized":"2024-04-24T23:50:20.788341+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T03:54:26.194711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788086+0000","last_change":"2024-04-24T23:50:19.083773+0000","last_active":"2024-04-24T23:50:20.788086+0000","last_peered":"2024-04-24T23:50:20.788086+0000","last_clean":"2024-04-24T23:50:20.788086+0000","last_became_active":"2024-04-24T23:50:19.083351+0000","last_became_peered":"2024-04-24T23:50:19.083351+0000","last_unstale":"2024-04-24T23:50:20.788086+0000","last_undegraded":"2024-04-24T23:50:20.788086+0000","last_fullsized":"2024-04-24T23:50:20.788086+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:42:41.658151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074811+0000","last_change":"2024-04-24T23:50:18.082274+0000","last_active":"2024-04-24T23:50:20.074811+0000","last_peered":"2024-04-24T23:50:20.074811+0000","last_clean":"2024-04-24T23:50:20.074811+0000","last_became_active":"2024-04-24T23:50:18.082002+0000","last_became_peered":"2024-04-24T23:50:18.082002+0000","last_unstale":"2024-04-24T23:50:20.074811+0000","last_undegraded":"2024-04-24T23:50:20.074811+0000","last_fullsized":"2024-04-24T23:50:20.074811+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T08:23:27.985620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788282+0000","last_change":"2024-04-24T23:50:19.092660+0000","last_active":"2024-04-24T23:50:20.788282+0000","last_peered":"2024-04-24T23:50:20.788282+0000","last_clean":"2024-04-24T23:50:20.788282+0000","last_became_active":"2024-04-24T23:50:19.091442+0000","last_became_peered":"2024-04-24T23:50:19.091442+0000","last_unstale":"2024-04-24T23:50:20.788282+0000","last_undegraded":"2024-04-24T23:50:20.788282+0000","last_fullsized":"2024-04-24T23:50:20.788282+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:20:53.860893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788113+0000","last_change":"2024-04-24T23:50:18.084654+0000","last_active":"2024-04-24T23:50:20.788113+0000","last_peered":"2024-04-24T23:50:20.788113+0000","last_clean":"2024-04-24T23:50:20.788113+0000","last_became_active":"2024-04-24T23:50:18.084408+0000","last_became_peered":"2024-04-24T23:50:18.084408+0000","last_unstale":"2024-04-24T23:50:20.788113+0000","last_undegraded":"2024-04-24T23:50:20.788113+0000","last_fullsized":"2024-04-24T23:50:20.788113+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:27:58.919581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.071936+0000","last_change":"2024-04-24T23:50:19.081808+0000","last_active":"2024-04-24T23:50:20.071936+0000","last_peered":"2024-04-24T23:50:20.071936+0000","last_clean":"2024-04-24T23:50:20.071936+0000","last_became_active":"2024-04-24T23:50:19.081538+0000","last_became_peered":"2024-04-24T23:50:19.081538+0000","last_unstale":"2024-04-24T23:50:20.071936+0000","last_undegraded":"2024-04-24T23:50:20.071936+0000","last_fullsized":"2024-04-24T23:50:20.071936+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T10:43:10.966605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"38'2","reported_seq":16,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788462+0000","last_change":"2024-04-24T23:50:18.089751+0000","last_active":"2024-04-24T23:50:20.788462+0000","last_peered":"2024-04-24T23:50:20.788462+0000","last_clean":"2024-04-24T23:50:20.788462+0000","last_became_active":"2024-04-24T23:50:18.089446+0000","last_became_peered":"2024-04-24T23:50:18.089446+0000","last_unstale":"2024-04-24T23:50:20.788462+0000","last_undegraded":"2024-04-24T23:50:20.788462+0000","last_fullsized":"2024-04-24T23:50:20.788462+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:39:00.830956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075154+0000","last_change":"2024-04-24T23:50:19.084843+0000","last_active":"2024-04-24T23:50:20.075154+0000","last_peered":"2024-04-24T23:50:20.075154+0000","last_clean":"2024-04-24T23:50:20.075154+0000","last_became_active":"2024-04-24T23:50:19.084536+0000","last_became_peered":"2024-04-24T23:50:19.084536+0000","last_unstale":"2024-04-24T23:50:20.075154+0000","last_undegraded":"2024-04-24T23:50:20.075154+0000","last_fullsized":"2024-04-24T23:50:20.075154+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:43:44.423218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"39'1","reported_seq":15,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788632+0000","last_change":"2024-04-24T23:50:18.077055+0000","last_active":"2024-04-24T23:50:20.788632+0000","last_peered":"2024-04-24T23:50:20.788632+0000","last_clean":"2024-04-24T23:50:20.788632+0000","last_became_active":"2024-04-24T23:50:18.076771+0000","last_became_peered":"2024-04-24T23:50:18.076771+0000","last_unstale":"2024-04-24T23:50:20.788632+0000","last_undegraded":"2024-04-24T23:50:20.788632+0000","last_fullsized":"2024-04-24T23:50:20.788632+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T11:50:13.502204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788342+0000","last_change":"2024-04-24T23:50:19.094990+0000","last_active":"2024-04-24T23:50:20.788342+0000","last_peered":"2024-04-24T23:50:20.788342+0000","last_clean":"2024-04-24T23:50:20.788342+0000","last_became_active":"2024-04-24T23:50:19.094677+0000","last_became_peered":"2024-04-24T23:50:19.094677+0000","last_unstale":"2024-04-24T23:50:20.788342+0000","last_undegraded":"2024-04-24T23:50:20.788342+0000","last_fullsized":"2024-04-24T23:50:20.788342+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:18:59.007356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788295+0000","last_change":"2024-04-24T23:50:18.080646+0000","last_active":"2024-04-24T23:50:20.788295+0000","last_peered":"2024-04-24T23:50:20.788295+0000","last_clean":"2024-04-24T23:50:20.788295+0000","last_became_active":"2024-04-24T23:50:18.080375+0000","last_became_peered":"2024-04-24T23:50:18.080375+0000","last_unstale":"2024-04-24T23:50:20.788295+0000","last_undegraded":"2024-04-24T23:50:20.788295+0000","last_fullsized":"2024-04-24T23:50:20.788295+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:57:40.483306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788879+0000","last_change":"2024-04-24T23:50:19.087874+0000","last_active":"2024-04-24T23:50:20.788879+0000","last_peered":"2024-04-24T23:50:20.788879+0000","last_clean":"2024-04-24T23:50:20.788879+0000","last_became_active":"2024-04-24T23:50:19.087578+0000","last_became_peered":"2024-04-24T23:50:19.087578+0000","last_unstale":"2024-04-24T23:50:20.788879+0000","last_undegraded":"2024-04-24T23:50:20.788879+0000","last_fullsized":"2024-04-24T23:50:20.788879+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:21:53.415628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.789202+0000","last_change":"2024-04-24T23:50:18.084424+0000","last_active":"2024-04-24T23:50:20.789202+0000","last_peered":"2024-04-24T23:50:20.789202+0000","last_clean":"2024-04-24T23:50:20.789202+0000","last_became_active":"2024-04-24T23:50:18.084119+0000","last_became_peered":"2024-04-24T23:50:18.084119+0000","last_unstale":"2024-04-24T23:50:20.789202+0000","last_undegraded":"2024-04-24T23:50:20.789202+0000","last_fullsized":"2024-04-24T23:50:20.789202+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:19:42.318258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.071766+0000","last_change":"2024-04-24T23:50:19.093033+0000","last_active":"2024-04-24T23:50:20.071766+0000","last_peered":"2024-04-24T23:50:20.071766+0000","last_clean":"2024-04-24T23:50:20.071766+0000","last_became_active":"2024-04-24T23:50:19.092771+0000","last_became_peered":"2024-04-24T23:50:19.092771+0000","last_unstale":"2024-04-24T23:50:20.071766+0000","last_undegraded":"2024-04-24T23:50:20.071766+0000","last_fullsized":"2024-04-24T23:50:20.071766+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:10:39.114823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074641+0000","last_change":"2024-04-24T23:50:18.088182+0000","last_active":"2024-04-24T23:50:20.074641+0000","last_peered":"2024-04-24T23:50:20.074641+0000","last_clean":"2024-04-24T23:50:20.074641+0000","last_became_active":"2024-04-24T23:50:18.087899+0000","last_became_peered":"2024-04-24T23:50:18.087899+0000","last_unstale":"2024-04-24T23:50:20.074641+0000","last_undegraded":"2024-04-24T23:50:20.074641+0000","last_fullsized":"2024-04-24T23:50:20.074641+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T07:19:42.318258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075771+0000","last_change":"2024-04-24T23:50:19.090504+0000","last_active":"2024-04-24T23:50:20.075771+0000","last_peered":"2024-04-24T23:50:20.075771+0000","last_clean":"2024-04-24T23:50:20.075771+0000","last_became_active":"2024-04-24T23:50:19.088678+0000","last_became_peered":"2024-04-24T23:50:19.088678+0000","last_unstale":"2024-04-24T23:50:20.075771+0000","last_undegraded":"2024-04-24T23:50:20.075771+0000","last_fullsized":"2024-04-24T23:50:20.075771+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T06:39:01.832638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.075576+0000","last_change":"2024-04-24T23:50:18.088177+0000","last_active":"2024-04-24T23:50:20.075576+0000","last_peered":"2024-04-24T23:50:20.075576+0000","last_clean":"2024-04-24T23:50:20.075576+0000","last_became_active":"2024-04-24T23:50:18.087915+0000","last_became_peered":"2024-04-24T23:50:18.087915+0000","last_unstale":"2024-04-24T23:50:20.075576+0000","last_undegraded":"2024-04-24T23:50:20.075576+0000","last_fullsized":"2024-04-24T23:50:20.075576+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:26:34.409949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074979+0000","last_change":"2024-04-24T23:50:19.080121+0000","last_active":"2024-04-24T23:50:20.074979+0000","last_peered":"2024-04-24T23:50:20.074979+0000","last_clean":"2024-04-24T23:50:20.074979+0000","last_became_active":"2024-04-24T23:50:19.079833+0000","last_became_peered":"2024-04-24T23:50:19.079833+0000","last_unstale":"2024-04-24T23:50:20.074979+0000","last_undegraded":"2024-04-24T23:50:20.074979+0000","last_fullsized":"2024-04-24T23:50:20.074979+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:42:58.003442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.787914+0000","last_change":"2024-04-24T23:50:18.090442+0000","last_active":"2024-04-24T23:50:20.787914+0000","last_peered":"2024-04-24T23:50:20.787914+0000","last_clean":"2024-04-24T23:50:20.787914+0000","last_became_active":"2024-04-24T23:50:18.090120+0000","last_became_peered":"2024-04-24T23:50:18.090120+0000","last_unstale":"2024-04-24T23:50:20.787914+0000","last_undegraded":"2024-04-24T23:50:20.787914+0000","last_fullsized":"2024-04-24T23:50:20.787914+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T00:39:49.970108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.788385+0000","last_change":"2024-04-24T23:50:19.092659+0000","last_active":"2024-04-24T23:50:20.788385+0000","last_peered":"2024-04-24T23:50:20.788385+0000","last_clean":"2024-04-24T23:50:20.788385+0000","last_became_active":"2024-04-24T23:50:19.092380+0000","last_became_peered":"2024-04-24T23:50:19.092380+0000","last_unstale":"2024-04-24T23:50:20.788385+0000","last_undegraded":"2024-04-24T23:50:20.788385+0000","last_fullsized":"2024-04-24T23:50:20.788385+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:18.062426+0000","last_clean_scrub_stamp":"2024-04-24T23:50:18.062426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T04:50:19.118407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074996+0000","last_change":"2024-04-24T23:50:18.083990+0000","last_active":"2024-04-24T23:50:20.074996+0000","last_peered":"2024-04-24T23:50:20.074996+0000","last_clean":"2024-04-24T23:50:20.074996+0000","last_became_active":"2024-04-24T23:50:18.083695+0000","last_became_peered":"2024-04-24T23:50:18.083695+0000","last_unstale":"2024-04-24T23:50:20.074996+0000","last_undegraded":"2024-04-24T23:50:20.074996+0000","last_fullsized":"2024-04-24T23:50:20.074996+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":36,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:50:17.060744+0000","last_clean_scrub_stamp":"2024-04-24T23:50:17.060744+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T01:55:08.953626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":206,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-04-24T23:50:20.074318+0000","last_change":"2024-04-24T23:48:45.082459+0000","last_active":"2024-04-24T23:50:20.074318+0000","last_peered":"2024-04-24T23:50:20.074318+0000","last_clean":"2024-04-24T23:50:20.074318+0000","last_became_active":"2024-04-24T23:48:45.081495+0000","last_became_peered":"2024-04-24T23:48:45.081495+0000","last_unstale":"2024-04-24T23:50:20.074318+0000","last_undegraded":"2024-04-24T23:50:20.074318+0000","last_fullsized":"2024-04-24T23:50:20.074318+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_clean_scrub_stamp":"2024-04-24T23:48:43.440993+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:53:20.832511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":786757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":195289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574871,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":458752,"data_stored":194819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31628,"kb_used_data":448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340212,"statfs":{"total":96636764160,"available":96604377088,"internally_reserved":0,"allocated":458752,"data_stored":195311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574872,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31636,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340204,"statfs":{"total":96636764160,"available":96604368896,"internally_reserved":0,"allocated":466944,"data_stored":196353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574872,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":194777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574871,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31624,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340216,"statfs":{"total":96636764160,"available":96604381184,"internally_reserved":0,"allocated":454656,"data_stored":194797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574871,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32204,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339636,"statfs":{"total":96636764160,"available":96603787264,"internally_reserved":0,"allocated":1048576,"data_stored":785109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:50:26.871 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-24T23:50:26.872 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-04-24T23:50:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.015+0000 7fefab46d640 1 Processor -- start 2024-04-24T23:50:27.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.016+0000 7fefab46d640 1 -- start start 2024-04-24T23:50:27.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.016+0000 7fefab46d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fefa4154ba0 0x7fefa40b5030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:27.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.016+0000 7fefab46d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa415acd0 con 0x7fefa4072a30 2024-04-24T23:50:27.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.016+0000 7fefab46d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefa415afb0 con 0x7fefa40b3750 2024-04-24T23:50:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.016+0000 7fefab46d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fefa415b290 con 0x7fefa4154ba0 2024-04-24T23:50:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.017+0000 7fefa9209640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fefa4154ba0 0x7fefa40b5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.017+0000 7fefa9a0a640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fefa4072a30 0x7fefa40cd770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55882/0 (socket says 172.21.15.16:55882) 2024-04-24T23:50:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.017+0000 7fefa9a0a640 1 -- 172.21.15.16:0/152863557 learned_addr learned my addr 172.21.15.16:0/152863557 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.017+0000 7fefa9209640 1 -- 172.21.15.16:0/152863557 >> v1:172.21.15.156:6789/0 conn(0x7fefa40b3750 legacy=0x7fefa4072320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.017+0000 7fefa9209640 1 -- 172.21.15.16:0/152863557 >> v1:172.21.15.16:6789/0 conn(0x7fefa4072a30 legacy=0x7fefa40cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.017+0000 7fefa9209640 1 -- 172.21.15.16:0/152863557 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa415b570 con 0x7fefa4154ba0 2024-04-24T23:50:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.017+0000 7fefa3fff640 1 -- 172.21.15.16:0/152863557 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2048659129 0 0) 0x7fefa415acd0 con 0x7fefa4072a30 2024-04-24T23:50:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.018+0000 7fefa9209640 1 --2- 172.21.15.16:0/152863557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa4154ba0 0x7fefa40b5030 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0097a0 tx=0x7fef9c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d933a62c3a4acd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.018+0000 7fefa3fff640 1 -- 172.21.15.16:0/152863557 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c037070 con 0x7fefa4154ba0 2024-04-24T23:50:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.018+0000 7fefa3fff640 1 -- 172.21.15.16:0/152863557 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef9c032440 con 0x7fefa4154ba0 2024-04-24T23:50:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.018+0000 7fefa3fff640 1 -- 172.21.15.16:0/152863557 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c032810 con 0x7fefa4154ba0 2024-04-24T23:50:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.018+0000 7fefab46d640 1 -- 172.21.15.16:0/152863557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa4154ba0 msgr2=0x7fefa40b5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.018+0000 7fefab46d640 1 --2- 172.21.15.16:0/152863557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa4154ba0 0x7fefa40b5030 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0097a0 tx=0x7fef9c02a1e0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.018+0000 7fefab46d640 1 -- 172.21.15.16:0/152863557 shutdown_connections 2024-04-24T23:50:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.018+0000 7fefab46d640 1 --2- 172.21.15.16:0/152863557 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa4154ba0 0x7fefa40b5030 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.018+0000 7fefab46d640 1 -- 172.21.15.16:0/152863557 >> 172.21.15.16:0/152863557 conn(0x7fefa40cfc50 msgr2=0x7fefa40d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.018+0000 7fefab46d640 1 -- 172.21.15.16:0/152863557 shutdown_connections 2024-04-24T23:50:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.018+0000 7fefab46d640 1 -- 172.21.15.16:0/152863557 wait complete. 2024-04-24T23:50:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.019+0000 7fefab46d640 1 Processor -- start 2024-04-24T23:50:27.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.020+0000 7fefab46d640 1 -- start start 2024-04-24T23:50:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.020+0000 7fefab46d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa4072a30 0x7fefa40af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.020+0000 7fefab46d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefa40b3750 0x7fefa4149420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.020+0000 7fefab46d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefa4154ba0 0x7fefa4147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.020+0000 7fefab46d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fefa415bc10 con 0x7fefa4072a30 2024-04-24T23:50:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.020+0000 7fefab46d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fefa415ba90 con 0x7fefa4154ba0 2024-04-24T23:50:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.020+0000 7fefab46d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fefa40b5570 con 0x7fefa40b3750 2024-04-24T23:50:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.020+0000 7fefa9209640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa4072a30 0x7fefa40af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.020+0000 7fefa9209640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa4072a30 0x7fefa40af930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38040/0 (socket says 172.21.15.16:38040) 2024-04-24T23:50:27.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.020+0000 7fefa9209640 1 -- 172.21.15.16:0/941491035 learned_addr learned my addr 172.21.15.16:0/941491035 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.021+0000 7fefa9209640 1 -- 172.21.15.16:0/941491035 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefa40b3750 msgr2=0x7fefa4149420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.021+0000 7fefa8a08640 1 --2- 172.21.15.16:0/941491035 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefa40b3750 0x7fefa4149420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.021+0000 7fefa9209640 1 --2- 172.21.15.16:0/941491035 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefa40b3750 0x7fefa4149420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.021+0000 7fefa9209640 1 -- 172.21.15.16:0/941491035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefa4154ba0 msgr2=0x7fefa4147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.021+0000 7fefa9209640 1 --2- 172.21.15.16:0/941491035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefa4154ba0 0x7fefa4147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.021+0000 7fefa9209640 1 -- 172.21.15.16:0/941491035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefa4149960 con 0x7fefa4072a30 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.021+0000 7fefa8a08640 1 --2- 172.21.15.16:0/941491035 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefa40b3750 0x7fefa4149420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.021+0000 7fefa9209640 1 --2- 172.21.15.16:0/941491035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa4072a30 0x7fefa40af930 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fef9c000c00 tx=0x7fef9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.021+0000 7fefa1ffb640 1 -- 172.21.15.16:0/941491035 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c037040 con 0x7fefa4072a30 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.021+0000 7fefa1ffb640 1 -- 172.21.15.16:0/941491035 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef9c042070 con 0x7fefa4072a30 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.021+0000 7fefab46d640 1 -- 172.21.15.16:0/941491035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefa40afff0 con 0x7fefa4072a30 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.021+0000 7fefa1ffb640 1 -- 172.21.15.16:0/941491035 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef9c030a70 con 0x7fefa4072a30 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.021+0000 7fefab46d640 1 -- 172.21.15.16:0/941491035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefa40b05b0 con 0x7fefa4072a30 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.022+0000 7fefab46d640 1 -- 172.21.15.16:0/941491035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefa4068f90 con 0x7fefa4072a30 2024-04-24T23:50:27.023 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.023+0000 7fefa1ffb640 1 -- 172.21.15.16:0/941491035 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fef9c030c10 con 0x7fefa4072a30 2024-04-24T23:50:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.023+0000 7fefa1ffb640 1 --2- 172.21.15.16:0/941491035 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fef9006be60 0x7fef9006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.023+0000 7fefa1ffb640 1 -- 172.21.15.16:0/941491035 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fef9c0abaa0 con 0x7fefa4072a30 2024-04-24T23:50:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.023+0000 7fefa8a08640 1 --2- 172.21.15.16:0/941491035 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fef9006be60 0x7fef9006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:27.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.024+0000 7fefa8a08640 1 --2- 172.21.15.16:0/941491035 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fef9006be60 0x7fef9006e320 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fef98003eb0 tx=0x7fef98009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:27.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.027+0000 7fefa1ffb640 1 -- 172.21.15.16:0/941491035 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fef9c07a2c0 con 0x7fefa4072a30 2024-04-24T23:50:27.284 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.284+0000 7fefab46d640 1 -- 172.21.15.16:0/941491035 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7fefa40b5010 con 0x7fefa4072a30 2024-04-24T23:50:27.286 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.286+0000 7fefa1ffb640 1 -- 172.21.15.16:0/941491035 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7fef9c079bc0 con 0x7fefa4072a30 2024-04-24T23:50:27.286 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-04-24T23:50:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAsmilm81VpJhAAXQnTDv4ShV2nH4Tpv5oziw== 2024-04-24T23:50:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: caps mds = "allow" 2024-04-24T23:50:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: caps mon = "allow r" 2024-04-24T23:50:27.286 INFO:teuthology.orchestra.run.smithi016.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-04-24T23:50:27.286 INFO:teuthology.orchestra.run.smithi016.stderr:updated caps for client.0 2024-04-24T23:50:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.288+0000 7fefab46d640 1 -- 172.21.15.16:0/941491035 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fef9006be60 msgr2=0x7fef9006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.288+0000 7fefab46d640 1 --2- 172.21.15.16:0/941491035 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fef9006be60 0x7fef9006e320 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fef98003eb0 tx=0x7fef98009040 comp rx=0 tx=0).stop 2024-04-24T23:50:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.288+0000 7fefab46d640 1 -- 172.21.15.16:0/941491035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa4072a30 msgr2=0x7fefa40af930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.288+0000 7fefab46d640 1 --2- 172.21.15.16:0/941491035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa4072a30 0x7fefa40af930 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fef9c000c00 tx=0x7fef9c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.288 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.288+0000 7fefab46d640 1 -- 172.21.15.16:0/941491035 shutdown_connections 2024-04-24T23:50:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.288+0000 7fefab46d640 1 --2- 172.21.15.16:0/941491035 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fef9006be60 0x7fef9006e320 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.288+0000 7fefab46d640 1 --2- 172.21.15.16:0/941491035 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fefa4154ba0 0x7fefa4147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.288+0000 7fefab46d640 1 --2- 172.21.15.16:0/941491035 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fefa40b3750 0x7fefa4149420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.288+0000 7fefab46d640 1 --2- 172.21.15.16:0/941491035 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fefa4072a30 0x7fefa40af930 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.288+0000 7fefab46d640 1 -- 172.21.15.16:0/941491035 >> 172.21.15.16:0/941491035 conn(0x7fefa40cfc50 msgr2=0x7fefa406df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.288+0000 7fefab46d640 1 -- 172.21.15.16:0/941491035 shutdown_connections 2024-04-24T23:50:27.289 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.288+0000 7fefab46d640 1 -- 172.21.15.16:0/941491035 wait complete. 2024-04-24T23:50:27.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:50:27.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.400+0000 7ff1e959e640 1 Processor -- start 2024-04-24T23:50:27.399 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.400+0000 7ff1e959e640 1 -- start start 2024-04-24T23:50:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.400+0000 7ff1e959e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1e406bf90 0x7ff1e41618d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.400+0000 7ff1e959e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1e4161e10 0x7ff1e40d24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.400+0000 7ff1e959e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1e415e1d0 con 0x7ff1e415ddf0 2024-04-24T23:50:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.400+0000 7ff1e959e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff1e4162260 con 0x7ff1e4161e10 2024-04-24T23:50:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.400+0000 7ff1e959e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e406cb30 con 0x7ff1e406bf90 2024-04-24T23:50:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.400+0000 7ff1e8d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1e4161e10 0x7ff1e40d24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.401+0000 7ff1e3fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff1e415ddf0 0x7ff1e40b2fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55888/0 (socket says 172.21.15.16:55888) 2024-04-24T23:50:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.401+0000 7ff1e3fff640 1 -- 172.21.15.16:0/701906477 learned_addr learned my addr 172.21.15.16:0/701906477 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.401+0000 7ff1e37fe640 1 --2- 172.21.15.16:0/701906477 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1e406bf90 0x7ff1e41618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.401+0000 7ff1e8d9d640 1 -- 172.21.15.16:0/701906477 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1e406bf90 msgr2=0x7ff1e41618d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:27.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.401+0000 7ff1e8d9d640 1 --2- 172.21.15.16:0/701906477 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1e406bf90 0x7ff1e41618d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.401+0000 7ff1e8d9d640 1 -- 172.21.15.16:0/701906477 >> v1:172.21.15.16:6789/0 conn(0x7ff1e415ddf0 legacy=0x7ff1e40b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:27.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.401+0000 7ff1e8d9d640 1 -- 172.21.15.16:0/701906477 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e406c570 con 0x7ff1e4161e10 2024-04-24T23:50:27.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.401+0000 7ff1e2ffd640 1 -- 172.21.15.16:0/701906477 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3191297652 0 0) 0x7ff1e415e1d0 con 0x7ff1e415ddf0 2024-04-24T23:50:27.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.402+0000 7ff1e8d9d640 1 --2- 172.21.15.16:0/701906477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e4161e10 0x7ff1e40d24e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc009d90 tx=0x7ff1dc02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1540a21ab7a30442 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:27.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.402+0000 7ff1e2ffd640 1 -- 172.21.15.16:0/701906477 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc031600 con 0x7ff1e4161e10 2024-04-24T23:50:27.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.402+0000 7ff1e2ffd640 1 -- 172.21.15.16:0/701906477 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1dc0317a0 con 0x7ff1e4161e10 2024-04-24T23:50:27.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.402+0000 7ff1e2ffd640 1 -- 172.21.15.16:0/701906477 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc031b70 con 0x7ff1e4161e10 2024-04-24T23:50:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.402+0000 7ff1e959e640 1 -- 172.21.15.16:0/701906477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e4161e10 msgr2=0x7ff1e40d24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.402+0000 7ff1e959e640 1 --2- 172.21.15.16:0/701906477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e4161e10 0x7ff1e40d24e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc009d90 tx=0x7ff1dc02a5d0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.402+0000 7ff1e959e640 1 -- 172.21.15.16:0/701906477 shutdown_connections 2024-04-24T23:50:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.402+0000 7ff1e959e640 1 --2- 172.21.15.16:0/701906477 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e4161e10 0x7ff1e40d24e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.402+0000 7ff1e959e640 1 --2- 172.21.15.16:0/701906477 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1e406bf90 0x7ff1e41618d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.402+0000 7ff1e959e640 1 -- 172.21.15.16:0/701906477 >> 172.21.15.16:0/701906477 conn(0x7ff1e40d4060 msgr2=0x7ff1e40d4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.402+0000 7ff1e959e640 1 -- 172.21.15.16:0/701906477 shutdown_connections 2024-04-24T23:50:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.402+0000 7ff1e959e640 1 -- 172.21.15.16:0/701906477 wait complete. 2024-04-24T23:50:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.404+0000 7ff1e959e640 1 Processor -- start 2024-04-24T23:50:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.404+0000 7ff1e959e640 1 -- start start 2024-04-24T23:50:27.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e959e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e406bf90 0x7ff1e415cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:27.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e959e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1e415ddf0 0x7ff1e40c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:27.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e959e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1e4161e10 0x7ff1e41f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:27.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e959e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e40d1ce0 con 0x7ff1e406bf90 2024-04-24T23:50:27.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e959e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e40d2050 con 0x7ff1e4161e10 2024-04-24T23:50:27.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e959e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff1e40b6e30 con 0x7ff1e415ddf0 2024-04-24T23:50:27.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e406bf90 0x7ff1e415cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e406bf90 0x7ff1e415cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38058/0 (socket says 172.21.15.16:38058) 2024-04-24T23:50:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e3fff640 1 -- 172.21.15.16:0/4088632305 learned_addr learned my addr 172.21.15.16:0/4088632305 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e37fe640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1e415ddf0 0x7ff1e40c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e8d9d640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1e4161e10 0x7ff1e41f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e3fff640 1 -- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1e415ddf0 msgr2=0x7ff1e40c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e3fff640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1e415ddf0 0x7ff1e40c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e3fff640 1 -- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1e4161e10 msgr2=0x7ff1e41f6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e3fff640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1e4161e10 0x7ff1e41f6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e3fff640 1 -- 172.21.15.16:0/4088632305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e415d060 con 0x7ff1e406bf90 2024-04-24T23:50:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e8d9d640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1e4161e10 0x7ff1e41f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e3fff640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e406bf90 0x7ff1e415cb20 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff1d400c500 tx=0x7ff1d400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e0ff9640 1 -- 172.21.15.16:0/4088632305 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4018070 con 0x7ff1e406bf90 2024-04-24T23:50:27.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e0ff9640 1 -- 172.21.15.16:0/4088632305 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1d4007580 con 0x7ff1e406bf90 2024-04-24T23:50:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e959e640 1 -- 172.21.15.16:0/4088632305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e40c9530 con 0x7ff1e406bf90 2024-04-24T23:50:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e0ff9640 1 -- 172.21.15.16:0/4088632305 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4004a20 con 0x7ff1e406bf90 2024-04-24T23:50:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.405+0000 7ff1e959e640 1 -- 172.21.15.16:0/4088632305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e40c9af0 con 0x7ff1e406bf90 2024-04-24T23:50:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.407+0000 7ff1e959e640 1 -- 172.21.15.16:0/4088632305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1e404f500 con 0x7ff1e406bf90 2024-04-24T23:50:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.407+0000 7ff1e0ff9640 1 -- 172.21.15.16:0/4088632305 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff1d401b070 con 0x7ff1e406bf90 2024-04-24T23:50:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.407+0000 7ff1e0ff9640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff1d006be30 0x7ff1d006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.407+0000 7ff1e0ff9640 1 -- 172.21.15.16:0/4088632305 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7ff1d408a1a0 con 0x7ff1e406bf90 2024-04-24T23:50:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.407+0000 7ff1e37fe640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff1d006be30 0x7ff1d006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.408+0000 7ff1e37fe640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff1d006be30 0x7ff1d006e2f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff1e40b89e0 tx=0x7ff1d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:27.411 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.412+0000 7ff1e0ff9640 1 -- 172.21.15.16:0/4088632305 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff1d4058400 con 0x7ff1e406bf90 2024-04-24T23:50:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.631+0000 7ff1e959e640 1 -- 172.21.15.16:0/4088632305 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1e40ced40 con 0x7ff1e406bf90 2024-04-24T23:50:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.632+0000 7ff1e0ff9640 1 -- 172.21.15.16:0/4088632305 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5181 (secure 0 0 0) 0x7ff1d405c700 con 0x7ff1e406bf90 2024-04-24T23:50:27.632 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:27.632 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5299,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/448535992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":448535992},{"type":"v1","addr":"172.21.15.16:6835","nonce":448535992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5305,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1890687780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1890687780},{"type":"v1","addr":"172.21.15.156:6836","nonce":1890687780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5308,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2538147395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":2538147395},{"type":"v1","addr":"172.21.15.156:6837","nonce":2538147395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:50:18.442310+0000","modified":"2024-04-24T23:50:20.441592+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.16:6834/561619268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":561619268},{"type":"v1","addr":"172.21.15.16:6834","nonce":561619268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5490,"qdb_cluster":[5490]},"id":2}]} 2024-04-24T23:50:27.632 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 15 2024-04-24T23:50:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.634+0000 7ff1e959e640 1 -- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff1d006be30 msgr2=0x7ff1d006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.634+0000 7ff1e959e640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff1d006be30 0x7ff1d006e2f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff1e40b89e0 tx=0x7ff1d8008040 comp rx=0 tx=0).stop 2024-04-24T23:50:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.634+0000 7ff1e959e640 1 -- 172.21.15.16:0/4088632305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e406bf90 msgr2=0x7ff1e415cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.634+0000 7ff1e959e640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e406bf90 0x7ff1e415cb20 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7ff1d400c500 tx=0x7ff1d400c9c0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.634+0000 7ff1e959e640 1 -- 172.21.15.16:0/4088632305 shutdown_connections 2024-04-24T23:50:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.634+0000 7ff1e959e640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff1d006be30 0x7ff1d006e2f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.634+0000 7ff1e959e640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1e4161e10 0x7ff1e41f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.634+0000 7ff1e959e640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff1e415ddf0 0x7ff1e40c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.634+0000 7ff1e959e640 1 --2- 172.21.15.16:0/4088632305 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1e406bf90 0x7ff1e415cb20 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.634+0000 7ff1e959e640 1 -- 172.21.15.16:0/4088632305 >> 172.21.15.16:0/4088632305 conn(0x7ff1e40d4060 msgr2=0x7ff1e40cf470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.634+0000 7ff1e959e640 1 -- 172.21.15.16:0/4088632305 shutdown_connections 2024-04-24T23:50:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:27.634+0000 7ff1e959e640 1 -- 172.21.15.16:0/4088632305 wait complete. 2024-04-24T23:50:28.048 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T23:50:18.442310+0000', 'modified': '2024-04-24T23:50:20.441592+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5490}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5490': {'gid': 5490, 'name': 'a', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.16:6834/561619268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6832', 'nonce': 561619268}, {'type': 'v1', 'addr': '172.21.15.16:6834', 'nonce': 561619268}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5490, 'qdb_cluster': [5490]} 2024-04-24T23:50:28.048 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T23:50:28.048 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-24T23:50:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.154+0000 7f0984542640 1 Processor -- start 2024-04-24T23:50:28.154 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.154+0000 7f0984542640 1 -- start start 2024-04-24T23:50:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.155+0000 7f0984542640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f097c13eba0 0x7f097c071a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.155+0000 7f0984542640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f097c15f890 con 0x7f097c071f40 2024-04-24T23:50:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.155+0000 7f0984542640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f097c15fb50 con 0x7f097c06c110 2024-04-24T23:50:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.155+0000 7f0984542640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f097c06c560 con 0x7f097c13eba0 2024-04-24T23:50:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.155+0000 7f09822de640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f097c13eba0 0x7f097c071a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.155+0000 7f0981add640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f097c071f40 0x7f097c06ba00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55896/0 (socket says 172.21.15.16:55896) 2024-04-24T23:50:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.155+0000 7f0981add640 1 -- 172.21.15.16:0/2799889150 learned_addr learned my addr 172.21.15.16:0/2799889150 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.155+0000 7f09822de640 1 --2- 172.21.15.16:0/2799889150 >> v2:172.21.15.16:3300/0 conn(0x7f097c13eba0 0x7f097c071a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38060/0 (socket says 172.21.15.16:38060) 2024-04-24T23:50:28.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.155+0000 7f09822de640 1 -- 172.21.15.16:0/2799889150 >> v1:172.21.15.156:6789/0 conn(0x7f097c06c110 legacy=0x7f097c06dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.155+0000 7f09822de640 1 -- 172.21.15.16:0/2799889150 >> v1:172.21.15.16:6789/0 conn(0x7f097c071f40 legacy=0x7f097c06ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.155+0000 7f09822de640 1 -- 172.21.15.16:0/2799889150 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097c15fe30 con 0x7f097c13eba0 2024-04-24T23:50:28.156 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.155+0000 7f09812dc640 1 -- 172.21.15.16:0/2799889150 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1028780114 0 0) 0x7f097c15f890 con 0x7f097c071f40 2024-04-24T23:50:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.156+0000 7f09822de640 1 --2- 172.21.15.16:0/2799889150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c13eba0 0x7f097c071a00 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0974009d90 tx=0x7f097402a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ec9b9d32080747 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.156+0000 7f09812dc640 1 -- 172.21.15.16:0/2799889150 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974007740 con 0x7f097c13eba0 2024-04-24T23:50:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.156+0000 7f09812dc640 1 -- 172.21.15.16:0/2799889150 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09740078e0 con 0x7f097c13eba0 2024-04-24T23:50:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.156+0000 7f09812dc640 1 -- 172.21.15.16:0/2799889150 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974007cb0 con 0x7f097c13eba0 2024-04-24T23:50:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.157+0000 7f0984542640 1 -- 172.21.15.16:0/2799889150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c13eba0 msgr2=0x7f097c071a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.157+0000 7f0984542640 1 --2- 172.21.15.16:0/2799889150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c13eba0 0x7f097c071a00 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f0974009d90 tx=0x7f097402a6f0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.157+0000 7f0984542640 1 -- 172.21.15.16:0/2799889150 shutdown_connections 2024-04-24T23:50:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.157+0000 7f0984542640 1 --2- 172.21.15.16:0/2799889150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c13eba0 0x7f097c071a00 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.157+0000 7f0984542640 1 -- 172.21.15.16:0/2799889150 >> 172.21.15.16:0/2799889150 conn(0x7f097c0d0ac0 msgr2=0x7f097c0d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:28.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.157+0000 7f0984542640 1 -- 172.21.15.16:0/2799889150 shutdown_connections 2024-04-24T23:50:28.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.157+0000 7f0984542640 1 -- 172.21.15.16:0/2799889150 wait complete. 2024-04-24T23:50:28.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.158+0000 7f0984542640 1 Processor -- start 2024-04-24T23:50:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.158+0000 7f0984542640 1 -- start start 2024-04-24T23:50:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.159+0000 7f0984542640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c06c110 0x7f097c0b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.159+0000 7f0984542640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f097c071f40 0x7f097c1e6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.159+0000 7f0984542640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f097c13eba0 0x7f097c151c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.159+0000 7f0984542640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f097c13ef80 con 0x7f097c06c110 2024-04-24T23:50:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.159+0000 7f0984542640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f097c160480 con 0x7f097c071f40 2024-04-24T23:50:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.159+0000 7f0984542640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f097c06e1f0 con 0x7f097c13eba0 2024-04-24T23:50:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.159+0000 7f09822de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c06c110 0x7f097c0b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.159+0000 7f09822de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c06c110 0x7f097c0b31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38066/0 (socket says 172.21.15.16:38066) 2024-04-24T23:50:28.160 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.159+0000 7f09822de640 1 -- 172.21.15.16:0/3736781529 learned_addr learned my addr 172.21.15.16:0/3736781529 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:28.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.159+0000 7f09822de640 1 -- 172.21.15.16:0/3736781529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f097c13eba0 msgr2=0x7f097c151c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.159+0000 7f09822de640 1 --2- 172.21.15.16:0/3736781529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f097c13eba0 0x7f097c151c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.159+0000 7f09822de640 1 -- 172.21.15.16:0/3736781529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f097c071f40 msgr2=0x7f097c1e6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.159+0000 7f09822de640 1 --2- 172.21.15.16:0/3736781529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f097c071f40 0x7f097c1e6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.159+0000 7f09822de640 1 -- 172.21.15.16:0/3736781529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f097c0b3900 con 0x7f097c06c110 2024-04-24T23:50:28.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.160+0000 7f09822de640 1 --2- 172.21.15.16:0/3736781529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c06c110 0x7f097c0b31d0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0974000f80 tx=0x7f0974004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:28.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.160+0000 7f097affd640 1 -- 172.21.15.16:0/3736781529 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f097403f070 con 0x7f097c06c110 2024-04-24T23:50:28.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.160+0000 7f097affd640 1 -- 172.21.15.16:0/3736781529 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0974031d00 con 0x7f097c06c110 2024-04-24T23:50:28.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.160+0000 7f0984542640 1 -- 172.21.15.16:0/3736781529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f097c0b3be0 con 0x7f097c06c110 2024-04-24T23:50:28.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.160+0000 7f097affd640 1 -- 172.21.15.16:0/3736781529 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974003b90 con 0x7f097c06c110 2024-04-24T23:50:28.161 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.160+0000 7f0984542640 1 -- 172.21.15.16:0/3736781529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f097c1521d0 con 0x7f097c06c110 2024-04-24T23:50:28.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.162+0000 7f097affd640 1 -- 172.21.15.16:0/3736781529 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0974004030 con 0x7f097c06c110 2024-04-24T23:50:28.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.162+0000 7f097affd640 1 --2- 172.21.15.16:0/3736781529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f096806be60 0x7f096806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.162 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.162+0000 7f097affd640 1 -- 172.21.15.16:0/3736781529 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f09740ac3a0 con 0x7f097c06c110 2024-04-24T23:50:28.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.162+0000 7f0981add640 1 --2- 172.21.15.16:0/3736781529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f096806be60 0x7f096806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.163+0000 7f0981add640 1 --2- 172.21.15.16:0/3736781529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f096806be60 0x7f096806e320 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0970007fd0 tx=0x7f0970007900 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:28.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.163+0000 7f0987253740 1 Processor -- start 2024-04-24T23:50:28.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.163+0000 7f0987253740 1 -- start start 2024-04-24T23:50:28.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f0987253740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a008e50 0x56199a00cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f0987253740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561999fe0b60 0x56199a009230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f0987253740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a009770 0x56199a00c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f0987253740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561999f44f00 con 0x56199a008e50 2024-04-24T23:50:28.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f0987253740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561999eed750 con 0x56199a009770 2024-04-24T23:50:28.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f0987253740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x561999f40ec0 con 0x561999fe0b60 2024-04-24T23:50:28.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f09822de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a008e50 0x56199a00cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f09822de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a008e50 0x56199a00cc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38078/0 (socket says 172.21.15.16:38078) 2024-04-24T23:50:28.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f09822de640 1 -- 172.21.15.16:0/1595151885 learned_addr learned my addr 172.21.15.16:0/1595151885 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:28.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f0981add640 1 --2- 172.21.15.16:0/1595151885 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561999fe0b60 0x56199a009230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f0982adf640 1 --2- 172.21.15.16:0/1595151885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a009770 0x56199a00c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f09822de640 1 -- 172.21.15.16:0/1595151885 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561999fe0b60 msgr2=0x56199a009230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:28.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f09822de640 1 --2- 172.21.15.16:0/1595151885 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561999fe0b60 0x56199a009230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f09822de640 1 -- 172.21.15.16:0/1595151885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a009770 msgr2=0x56199a00c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:28.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f09822de640 1 --2- 172.21.15.16:0/1595151885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a009770 0x56199a00c000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f09822de640 1 -- 172.21.15.16:0/1595151885 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561999c592d0 con 0x56199a008e50 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f0982adf640 1 --2- 172.21.15.16:0/1595151885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a009770 0x56199a00c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.164+0000 7f09822de640 1 --2- 172.21.15.16:0/1595151885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a008e50 0x56199a00cc50 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f09740aa680 tx=0x7f0974007fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b761376bc235d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.165+0000 7f0963fff640 1 -- 172.21.15.16:0/1595151885 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974048070 con 0x56199a008e50 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.165+0000 7f0963fff640 1 -- 172.21.15.16:0/1595151885 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09740830a0 con 0x56199a008e50 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.165+0000 7f0963fff640 1 -- 172.21.15.16:0/1595151885 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0974078730 con 0x56199a008e50 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.165+0000 7f0987253740 1 -- 172.21.15.16:0/1595151885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a008e50 msgr2=0x56199a00cc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.165+0000 7f0987253740 1 --2- 172.21.15.16:0/1595151885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a008e50 0x56199a00cc50 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f09740aa680 tx=0x7f0974007fc0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.165+0000 7f0987253740 1 -- 172.21.15.16:0/1595151885 shutdown_connections 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.165+0000 7f0987253740 1 --2- 172.21.15.16:0/1595151885 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a009770 0x56199a00c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.165+0000 7f0987253740 1 --2- 172.21.15.16:0/1595151885 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561999fe0b60 0x56199a009230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.165+0000 7f0987253740 1 --2- 172.21.15.16:0/1595151885 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a008e50 0x56199a00cc50 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.165+0000 7f0987253740 1 -- 172.21.15.16:0/1595151885 >> 172.21.15.16:0/1595151885 conn(0x561999fde760 msgr2=0x56199a027000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.165+0000 7f0987253740 1 -- 172.21.15.16:0/1595151885 shutdown_connections 2024-04-24T23:50:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.165+0000 7f0987253740 1 -- 172.21.15.16:0/1595151885 wait complete. 2024-04-24T23:50:28.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.166+0000 7f0987253740 1 -- start start 2024-04-24T23:50:28.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.166+0000 7f0987253740 1 Processor -- start 2024-04-24T23:50:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0987253740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a008e50 0x561999ffccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0987253740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a009770 0x56199a007750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0987253740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a0002f0 0x56199a007c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0987253740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561999fdcab0 con 0x56199a009770 2024-04-24T23:50:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0987253740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561999f00cf0 con 0x56199a008e50 2024-04-24T23:50:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0987253740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x561999fdc4f0 con 0x56199a0002f0 2024-04-24T23:50:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0987253740 10 client.0 _ll_register_callbacks cb 0x561999f006e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:50:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0981add640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a009770 0x56199a007750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0981add640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a009770 0x56199a007750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38094/0 (socket says 172.21.15.16:38094) 2024-04-24T23:50:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0981add640 1 -- 172.21.15.16:0/3085859024 learned_addr learned my addr 172.21.15.16:0/3085859024 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f09822de640 1 --2- 172.21.15.16:0/3085859024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a008e50 0x561999ffccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0982adf640 1 --2- 172.21.15.16:0/3085859024 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a0002f0 0x56199a007c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0981add640 1 -- 172.21.15.16:0/3085859024 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a0002f0 msgr2=0x56199a007c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0981add640 1 --2- 172.21.15.16:0/3085859024 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a0002f0 0x56199a007c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0981add640 1 -- 172.21.15.16:0/3085859024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a008e50 msgr2=0x561999ffccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0981add640 1 --2- 172.21.15.16:0/3085859024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a008e50 0x561999ffccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0981add640 1 -- 172.21.15.16:0/3085859024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561999d236b0 con 0x56199a009770 2024-04-24T23:50:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.167+0000 7f0982adf640 1 --2- 172.21.15.16:0/3085859024 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a0002f0 0x56199a007c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f0981add640 1 --2- 172.21.15.16:0/3085859024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a009770 0x56199a007750 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f097c06c7a0 tx=0x7f0970001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f6864d27fc53e37 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f09617fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-24T23:50:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f09617fa640 1 -- 172.21.15.16:0/3085859024 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0970012440 con 0x56199a009770 2024-04-24T23:50:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f09617fa640 1 -- 172.21.15.16:0/3085859024 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f09700125e0 con 0x56199a009770 2024-04-24T23:50:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f0987253740 1 -- 172.21.15.16:0/3085859024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f09740830a0 con 0x56199a009770 2024-04-24T23:50:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f09617fa640 1 -- 172.21.15.16:0/3085859024 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f09700127e0 con 0x56199a009770 2024-04-24T23:50:28.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f0987253740 10 client.5709 fetch_fsmap learned FSMap version 15 2024-04-24T23:50:28.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f0987253740 1 -- 172.21.15.16:0/3085859024 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x56199a00acd0 con 0x56199a009770 2024-04-24T23:50:28.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f09617fa640 1 -- 172.21.15.16:0/3085859024 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09740830a0 con 0x56199a009770 2024-04-24T23:50:28.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f09617fa640 1 -- 172.21.15.16:0/3085859024 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7f097002b080 con 0x56199a009770 2024-04-24T23:50:28.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f0987253740 10 client.5709 fetch_fsmap finished waiting for FSMap version 15 2024-04-24T23:50:28.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f0987253740 10 client.5709 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T23:50:28.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f0987253740 20 client.5709 populate_metadata read hostname 'smithi016' 2024-04-24T23:50:28.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f0987253740 1 --2- 172.21.15.16:0/3085859024 >> [v2:172.21.15.16:6832/561619268,v1:172.21.15.16:6834/561619268] conn(0x56199a0477f0 0x56199a049be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f0987253740 4 client.5709 mds_command: new command op to 5490 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:50:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f0987253740 1 -- 172.21.15.16:0/3085859024 --> [v2:172.21.15.16:6832/561619268,v1:172.21.15.16:6834/561619268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561999f52440 con 0x56199a0477f0 2024-04-24T23:50:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.168+0000 7f09822de640 1 --2- 172.21.15.16:0/3085859024 >> [v2:172.21.15.16:6832/561619268,v1:172.21.15.16:6834/561619268] conn(0x56199a0477f0 0x56199a049be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.169+0000 7f09617fa640 10 client.5709 ms_handle_connect on v2:172.21.15.16:6832/561619268 2024-04-24T23:50:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.169+0000 7f09822de640 1 --2- 172.21.15.16:0/3085859024 >> [v2:172.21.15.16:6832/561619268,v1:172.21.15.16:6834/561619268] conn(0x56199a0477f0 0x56199a049be0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29585c7f2ae0676b server_cookie=c38f623766b7e1fb in_seq=0 out_seq=0 2024-04-24T23:50:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.169+0000 7f09617fa640 1 -- 172.21.15.16:0/3085859024 <== mds.0 v2:172.21.15.16:6832/561619268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561999f52440 con 0x56199a0477f0 2024-04-24T23:50:28.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.169+0000 7f09617fa640 10 client.5709 handle_command_reply: tid=0 2024-04-24T23:50:28.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.169+0000 7f0987253740 1 client.5709 shutdown 2024-04-24T23:50:28.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.169+0000 7f0987253740 1 -- 172.21.15.16:0/3085859024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a009770 msgr2=0x56199a007750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.169+0000 7f0987253740 1 --2- 172.21.15.16:0/3085859024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a009770 0x56199a007750 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f097c06c7a0 tx=0x7f0970001080 comp rx=0 tx=0).stop 2024-04-24T23:50:28.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.169+0000 7f0987253740 1 -- 172.21.15.16:0/3085859024 shutdown_connections 2024-04-24T23:50:28.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.170+0000 7f0987253740 1 --2- 172.21.15.16:0/3085859024 >> [v2:172.21.15.16:6832/561619268,v1:172.21.15.16:6834/561619268] conn(0x56199a0477f0 0x56199a049be0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.170+0000 7f0987253740 1 --2- 172.21.15.16:0/3085859024 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a0002f0 0x56199a007c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.170+0000 7f0987253740 1 --2- 172.21.15.16:0/3085859024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a009770 0x56199a007750 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.170+0000 7f09617fa640 0 client.5709 ms_handle_reset on v2:172.21.15.16:6832/561619268 2024-04-24T23:50:28.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.170+0000 7f0987253740 1 --2- 172.21.15.16:0/3085859024 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a008e50 0x561999ffccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.170+0000 7f0987253740 1 -- 172.21.15.16:0/3085859024 >> 172.21.15.16:0/3085859024 conn(0x561999fe0b60 msgr2=0x561999fe2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:28.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.170+0000 7f0987253740 1 -- 172.21.15.16:0/3085859024 shutdown_connections 2024-04-24T23:50:28.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.170+0000 7f0987253740 1 -- 172.21.15.16:0/3085859024 wait complete. 2024-04-24T23:50:28.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.170+0000 7f0987253740 20 client.5709 trim_cache size 0 max 16384 2024-04-24T23:50:28.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.188+0000 7f0987253740 1 Processor -- start 2024-04-24T23:50:28.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.188+0000 7f0987253740 1 -- start start 2024-04-24T23:50:28.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.188+0000 7f0987253740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a0002f0 0x561999fd78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f0987253740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a008e50 0x561999fe2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f0987253740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a009770 0x561999fe3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f0987253740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f097c0cf910 con 0x56199a0002f0 2024-04-24T23:50:28.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f0987253740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f097c1f2c90 con 0x56199a008e50 2024-04-24T23:50:28.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f0987253740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f097c1f32e0 con 0x56199a009770 2024-04-24T23:50:28.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f09822de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a0002f0 0x561999fd78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f09822de640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a0002f0 0x561999fd78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38098/0 (socket says 172.21.15.16:38098) 2024-04-24T23:50:28.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f09822de640 1 -- 172.21.15.16:0/1597493640 learned_addr learned my addr 172.21.15.16:0/1597493640 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:28.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f09822de640 1 -- 172.21.15.16:0/1597493640 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a009770 msgr2=0x561999fe3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f09822de640 1 --2- 172.21.15.16:0/1597493640 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a009770 0x561999fe3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f09822de640 1 -- 172.21.15.16:0/1597493640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a008e50 msgr2=0x561999fe2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f09822de640 1 --2- 172.21.15.16:0/1597493640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a008e50 0x561999fe2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f09822de640 1 -- 172.21.15.16:0/1597493640 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561999fe12b0 con 0x56199a0002f0 2024-04-24T23:50:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f09822de640 1 --2- 172.21.15.16:0/1597493640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a0002f0 0x561999fd78a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f097402ace0 tx=0x7f097407c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137e6747d1640876 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f09637fe640 1 -- 172.21.15.16:0/1597493640 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09740830a0 con 0x56199a0002f0 2024-04-24T23:50:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f09637fe640 1 -- 172.21.15.16:0/1597493640 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0974096070 con 0x56199a0002f0 2024-04-24T23:50:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.189+0000 7f09637fe640 1 -- 172.21.15.16:0/1597493640 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09740aa380 con 0x56199a0002f0 2024-04-24T23:50:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.190+0000 7f0987253740 1 -- 172.21.15.16:0/1597493640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a0002f0 msgr2=0x561999fd78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.190+0000 7f0987253740 1 --2- 172.21.15.16:0/1597493640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a0002f0 0x561999fd78a0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f097402ace0 tx=0x7f097407c000 comp rx=0 tx=0).stop 2024-04-24T23:50:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.190+0000 7f0987253740 1 -- 172.21.15.16:0/1597493640 shutdown_connections 2024-04-24T23:50:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.190+0000 7f0987253740 1 --2- 172.21.15.16:0/1597493640 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a009770 0x561999fe3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.190+0000 7f0987253740 1 --2- 172.21.15.16:0/1597493640 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a008e50 0x561999fe2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.190+0000 7f0987253740 1 --2- 172.21.15.16:0/1597493640 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a0002f0 0x561999fd78a0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.190+0000 7f0987253740 1 -- 172.21.15.16:0/1597493640 >> 172.21.15.16:0/1597493640 conn(0x561999fe0b60 msgr2=0x561999ffbe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:28.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.190+0000 7f0987253740 1 -- 172.21.15.16:0/1597493640 shutdown_connections 2024-04-24T23:50:28.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.190+0000 7f0987253740 1 -- 172.21.15.16:0/1597493640 wait complete. 2024-04-24T23:50:28.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.190+0000 7f0987253740 1 -- start start 2024-04-24T23:50:28.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.190+0000 7f0987253740 1 Processor -- start 2024-04-24T23:50:28.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.191+0000 7f0987253740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a0002f0 0x56199a0a4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.196 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.191+0000 7f0987253740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a008e50 0x561999fe2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.191+0000 7f0987253740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a009770 0x561999fe3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.191+0000 7f0987253740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x56199a008310 con 0x56199a008e50 2024-04-24T23:50:28.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.191+0000 7f0987253740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x56199a00af90 con 0x56199a009770 2024-04-24T23:50:28.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.191+0000 7f0987253740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x56199a029640 con 0x56199a0002f0 2024-04-24T23:50:28.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.191+0000 7f0987253740 10 client.0 _ll_register_callbacks cb 0x7f0970005a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:50:28.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.191+0000 7f0981add640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a008e50 0x561999fe2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.191+0000 7f0981add640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a008e50 0x561999fe2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38108/0 (socket says 172.21.15.16:38108) 2024-04-24T23:50:28.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.191+0000 7f0981add640 1 -- 172.21.15.16:0/1575924936 learned_addr learned my addr 172.21.15.16:0/1575924936 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:28.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.191+0000 7f0982adf640 1 --2- 172.21.15.16:0/1575924936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a009770 0x561999fe3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.191+0000 7f09822de640 1 --2- 172.21.15.16:0/1575924936 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a0002f0 0x56199a0a4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f0981add640 1 -- 172.21.15.16:0/1575924936 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a0002f0 msgr2=0x56199a0a4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:28.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f0981add640 1 --2- 172.21.15.16:0/1575924936 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a0002f0 0x56199a0a4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f0981add640 1 -- 172.21.15.16:0/1575924936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a009770 msgr2=0x561999fe3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:28.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f0981add640 1 --2- 172.21.15.16:0/1575924936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a009770 0x561999fe3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f0981add640 1 -- 172.21.15.16:0/1575924936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561999ffee60 con 0x56199a008e50 2024-04-24T23:50:28.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f0981add640 1 --2- 172.21.15.16:0/1575924936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a008e50 0x561999fe2f40 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f097c06c7a0 tx=0x7f0970000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d30cb4c2824b246 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:28.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f09617fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-24T23:50:28.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f09617fa640 1 -- 172.21.15.16:0/1575924936 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0970012980 con 0x56199a008e50 2024-04-24T23:50:28.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f09617fa640 1 -- 172.21.15.16:0/1575924936 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f0970022470 con 0x56199a008e50 2024-04-24T23:50:28.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f0987253740 1 -- 172.21.15.16:0/1575924936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0974096070 con 0x56199a008e50 2024-04-24T23:50:28.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f09617fa640 1 -- 172.21.15.16:0/1575924936 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7f097003a050 con 0x56199a008e50 2024-04-24T23:50:28.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f0987253740 10 client.5715 fetch_fsmap learned FSMap version 15 2024-04-24T23:50:28.198 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f0987253740 1 -- 172.21.15.16:0/1575924936 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x56199a0a55e0 con 0x56199a008e50 2024-04-24T23:50:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f09617fa640 1 -- 172.21.15.16:0/1575924936 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0974096070 con 0x56199a008e50 2024-04-24T23:50:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.192+0000 7f09617fa640 1 -- 172.21.15.16:0/1575924936 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7f097003d020 con 0x56199a008e50 2024-04-24T23:50:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 10 client.5715 fetch_fsmap finished waiting for FSMap version 15 2024-04-24T23:50:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 10 client.5715 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T23:50:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 20 client.5715 populate_metadata read hostname 'smithi016' 2024-04-24T23:50:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 1 --2- 172.21.15.16:0/1575924936 >> [v2:172.21.15.16:6832/561619268,v1:172.21.15.16:6834/561619268] conn(0x56199a026fa0 0x56199a0b0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 4 client.5715 mds_command: new command op to 5490 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:50:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 1 -- 172.21.15.16:0/1575924936 --> [v2:172.21.15.16:6832/561619268,v1:172.21.15.16:6834/561619268] -- command(tid 0: {"prefix": "status"}) -- 0x561999f4c790 con 0x56199a026fa0 2024-04-24T23:50:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f09822de640 1 --2- 172.21.15.16:0/1575924936 >> [v2:172.21.15.16:6832/561619268,v1:172.21.15.16:6834/561619268] conn(0x56199a026fa0 0x56199a0b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f09617fa640 10 client.5715 ms_handle_connect on v2:172.21.15.16:6832/561619268 2024-04-24T23:50:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f09822de640 1 --2- 172.21.15.16:0/1575924936 >> [v2:172.21.15.16:6832/561619268,v1:172.21.15.16:6834/561619268] conn(0x56199a026fa0 0x56199a0b0980 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cac1f769ddcc587 server_cookie=6732bd99cbb20cf8 in_seq=0 out_seq=0 2024-04-24T23:50:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f09617fa640 1 -- 172.21.15.16:0/1575924936 <== mds.0 v2:172.21.15.16:6832/561619268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x561999f4c790 con 0x56199a026fa0 2024-04-24T23:50:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f09617fa640 10 client.5715 handle_command_reply: tid=0 2024-04-24T23:50:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 1 client.5715 shutdown 2024-04-24T23:50:28.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 1 -- 172.21.15.16:0/1575924936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a008e50 msgr2=0x561999fe2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 1 --2- 172.21.15.16:0/1575924936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a008e50 0x561999fe2f40 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f097c06c7a0 tx=0x7f0970000ed0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 1 -- 172.21.15.16:0/1575924936 shutdown_connections 2024-04-24T23:50:28.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 1 --2- 172.21.15.16:0/1575924936 >> [v2:172.21.15.16:6832/561619268,v1:172.21.15.16:6834/561619268] conn(0x56199a026fa0 0x56199a0b0980 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 1 --2- 172.21.15.16:0/1575924936 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x56199a009770 0x561999fe3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 1 --2- 172.21.15.16:0/1575924936 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x56199a008e50 0x561999fe2f40 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f09617fa640 0 client.5715 ms_handle_reset on v2:172.21.15.16:6832/561619268 2024-04-24T23:50:28.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 1 --2- 172.21.15.16:0/1575924936 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x56199a0002f0 0x56199a0a4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.193+0000 7f0987253740 1 -- 172.21.15.16:0/1575924936 >> 172.21.15.16:0/1575924936 conn(0x561999fe0b60 msgr2=0x56199a007470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:28.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0987253740 1 -- 172.21.15.16:0/1575924936 shutdown_connections 2024-04-24T23:50:28.200 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0987253740 1 -- 172.21.15.16:0/1575924936 wait complete. 2024-04-24T23:50:28.201 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0987253740 20 client.5715 trim_cache size 0 max 16384 2024-04-24T23:50:28.201 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:50:28.201 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "f8999c8c-ab57-4aba-8d53-08e19e6ee3d5", 2024-04-24T23:50:28.201 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-04-24T23:50:28.201 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 5490, 2024-04-24T23:50:28.201 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-24T23:50:28.201 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-24T23:50:28.201 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-24T23:50:28.201 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 9.1252326830000001, 2024-04-24T23:50:28.201 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 15, 2024-04-24T23:50:28.202 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 39, 2024-04-24T23:50:28.202 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 39, 2024-04-24T23:50:28.202 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 15.987789027 2024-04-24T23:50:28.202 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:50:28.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0984542640 1 -- 172.21.15.16:0/3736781529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f096806be60 msgr2=0x7f096806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0984542640 1 --2- 172.21.15.16:0/3736781529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f096806be60 0x7f096806e320 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f0970007fd0 tx=0x7f0970007900 comp rx=0 tx=0).stop 2024-04-24T23:50:28.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0984542640 1 -- 172.21.15.16:0/3736781529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c06c110 msgr2=0x7f097c0b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0984542640 1 --2- 172.21.15.16:0/3736781529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c06c110 0x7f097c0b31d0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f0974000f80 tx=0x7f0974004a40 comp rx=0 tx=0).stop 2024-04-24T23:50:28.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0984542640 1 -- 172.21.15.16:0/3736781529 shutdown_connections 2024-04-24T23:50:28.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0984542640 1 --2- 172.21.15.16:0/3736781529 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f096806be60 0x7f096806e320 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0984542640 1 --2- 172.21.15.16:0/3736781529 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f097c13eba0 0x7f097c151c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0984542640 1 --2- 172.21.15.16:0/3736781529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f097c071f40 0x7f097c1e6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0984542640 1 --2- 172.21.15.16:0/3736781529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f097c06c110 0x7f097c0b31d0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0984542640 1 -- 172.21.15.16:0/3736781529 >> 172.21.15.16:0/3736781529 conn(0x7f097c0d0ac0 msgr2=0x7f097c06e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0984542640 1 -- 172.21.15.16:0/3736781529 shutdown_connections 2024-04-24T23:50:28.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.194+0000 7f0984542640 1 -- 172.21.15.16:0/3736781529 wait complete. 2024-04-24T23:50:28.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-04-24T23:50:28.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.306+0000 7fbf922ba640 1 Processor -- start 2024-04-24T23:50:28.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.306+0000 7fbf922ba640 1 -- start start 2024-04-24T23:50:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.306+0000 7fbf922ba640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbf8c0ce5f0 0x7fbf8c0cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf922ba640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf8c0ccad0 0x7fbf8c0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf922ba640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf8c0d05a0 con 0x7fbf8c15dfa0 2024-04-24T23:50:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf922ba640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fbf8c0b7360 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf922ba640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fbf8c0cc640 con 0x7fbf8c0ccad0 2024-04-24T23:50:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf8b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbf8c0ce5f0 0x7fbf8c0cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf90857640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf8c0ccad0 0x7fbf8c0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf8b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbf8c0ce5f0 0x7fbf8c0cea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:53940/0 (socket says 172.21.15.16:53940) 2024-04-24T23:50:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf90857640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf8c0ccad0 0x7fbf8c0ccf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:41344/0 (socket says 172.21.15.16:41344) 2024-04-24T23:50:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf8b7fe640 1 -- 172.21.15.16:0/3102978728 learned_addr learned my addr 172.21.15.16:0/3102978728 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf8bfff640 1 --1- 172.21.15.16:0/3102978728 >> v1:172.21.15.156:6789/0 conn(0x7fbf8c15dfa0 0x7fbf8c0d0fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:56244/0 (socket says 172.21.15.16:56244) 2024-04-24T23:50:28.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf8b7fe640 1 -- 172.21.15.16:0/3102978728 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf8c0ccad0 msgr2=0x7fbf8c0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf8b7fe640 1 --2- 172.21.15.16:0/3102978728 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf8c0ccad0 0x7fbf8c0ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf8b7fe640 1 -- 172.21.15.16:0/3102978728 >> v1:172.21.15.156:6789/0 conn(0x7fbf8c15dfa0 legacy=0x7fbf8c0d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf8b7fe640 1 -- 172.21.15.16:0/3102978728 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8c0d0880 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.307+0000 7fbf8affd640 1 -- 172.21.15.16:0/3102978728 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1872018181 0 0) 0x7fbf8c0d05a0 con 0x7fbf8c15dfa0 2024-04-24T23:50:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.308+0000 7fbf8b7fe640 1 --2- 172.21.15.16:0/3102978728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf8c0ce5f0 0x7fbf8c0cea70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009870 tx=0x7fbf8002a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=740b3c5b4b250ae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.308+0000 7fbf8affd640 1 -- 172.21.15.16:0/3102978728 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80037070 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.308+0000 7fbf8affd640 1 -- 172.21.15.16:0/3102978728 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf80032440 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.308+0000 7fbf8affd640 1 -- 172.21.15.16:0/3102978728 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80032810 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.309+0000 7fbf922ba640 1 -- 172.21.15.16:0/3102978728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf8c0ce5f0 msgr2=0x7fbf8c0cea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.309+0000 7fbf922ba640 1 --2- 172.21.15.16:0/3102978728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf8c0ce5f0 0x7fbf8c0cea70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009870 tx=0x7fbf8002a270 comp rx=0 tx=0).stop 2024-04-24T23:50:28.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.309+0000 7fbf922ba640 1 -- 172.21.15.16:0/3102978728 shutdown_connections 2024-04-24T23:50:28.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.309+0000 7fbf922ba640 1 --2- 172.21.15.16:0/3102978728 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf8c0ccad0 0x7fbf8c0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.309+0000 7fbf922ba640 1 --2- 172.21.15.16:0/3102978728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf8c0ce5f0 0x7fbf8c0cea70 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.309+0000 7fbf922ba640 1 -- 172.21.15.16:0/3102978728 >> 172.21.15.16:0/3102978728 conn(0x7fbf8c06b4d0 msgr2=0x7fbf8c0b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:28.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.309+0000 7fbf922ba640 1 -- 172.21.15.16:0/3102978728 shutdown_connections 2024-04-24T23:50:28.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.309+0000 7fbf922ba640 1 -- 172.21.15.16:0/3102978728 wait complete. 2024-04-24T23:50:28.312 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.310+0000 7fbf922ba640 1 Processor -- start 2024-04-24T23:50:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.310+0000 7fbf922ba640 1 -- start start 2024-04-24T23:50:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.311+0000 7fbf922ba640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf8c0ccad0 0x7fbf8c152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.311+0000 7fbf922ba640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf8c0ce5f0 0x7fbf8c15a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.311+0000 7fbf922ba640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf8c15dfa0 0x7fbf8c155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.311+0000 7fbf922ba640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c15e380 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.311+0000 7fbf922ba640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c0cc2d0 con 0x7fbf8c0ccad0 2024-04-24T23:50:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.311+0000 7fbf922ba640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fbf8c0d01c0 con 0x7fbf8c15dfa0 2024-04-24T23:50:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.311+0000 7fbf8b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf8c0ce5f0 0x7fbf8c15a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.311+0000 7fbf8b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf8c0ce5f0 0x7fbf8c15a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38118/0 (socket says 172.21.15.16:38118) 2024-04-24T23:50:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.311+0000 7fbf8b7fe640 1 -- 172.21.15.16:0/479137592 learned_addr learned my addr 172.21.15.16:0/479137592 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.311+0000 7fbf90857640 1 --2- 172.21.15.16:0/479137592 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf8c15dfa0 0x7fbf8c155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.311+0000 7fbf8bfff640 1 --2- 172.21.15.16:0/479137592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf8c0ccad0 0x7fbf8c152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.312+0000 7fbf8b7fe640 1 -- 172.21.15.16:0/479137592 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf8c15dfa0 msgr2=0x7fbf8c155950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.313 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.312+0000 7fbf8b7fe640 1 --2- 172.21.15.16:0/479137592 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf8c15dfa0 0x7fbf8c155950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.312+0000 7fbf8b7fe640 1 -- 172.21.15.16:0/479137592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf8c0ccad0 msgr2=0x7fbf8c152f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.312+0000 7fbf8b7fe640 1 --2- 172.21.15.16:0/479137592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf8c0ccad0 0x7fbf8c152f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.312+0000 7fbf8b7fe640 1 -- 172.21.15.16:0/479137592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8c154430 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.312+0000 7fbf8b7fe640 1 --2- 172.21.15.16:0/479137592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf8c0ce5f0 0x7fbf8c15a340 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbf80035040 tx=0x7fbf800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.312+0000 7fbf88ff9640 1 -- 172.21.15.16:0/479137592 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80037040 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.312+0000 7fbf88ff9640 1 -- 172.21.15.16:0/479137592 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf80004030 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.312+0000 7fbf922ba640 1 -- 172.21.15.16:0/479137592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8c154710 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.312+0000 7fbf88ff9640 1 -- 172.21.15.16:0/479137592 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf80004420 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.312+0000 7fbf922ba640 1 -- 172.21.15.16:0/479137592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf8c154cd0 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.313+0000 7fbf922ba640 1 -- 172.21.15.16:0/479137592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf8c06bea0 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.313+0000 7fbf88ff9640 1 -- 172.21.15.16:0/479137592 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbf8003a070 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.314+0000 7fbf88ff9640 1 --2- 172.21.15.16:0/479137592 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbf7806be60 0x7fbf7806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.314+0000 7fbf88ff9640 1 -- 172.21.15.16:0/479137592 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7fbf800ab3b0 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.314+0000 7fbf8bfff640 1 --2- 172.21.15.16:0/479137592 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbf7806be60 0x7fbf7806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:28.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.314+0000 7fbf8bfff640 1 --2- 172.21.15.16:0/479137592 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbf7806be60 0x7fbf7806e320 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c000f60 tx=0x7fbf7c007d50 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:28.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.318+0000 7fbf88ff9640 1 -- 172.21.15.16:0/479137592 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbf80041020 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.542 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.543+0000 7fbf922ba640 1 -- 172.21.15.16:0/479137592 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbf8c0cbf90 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.543+0000 7fbf88ff9640 1 -- 172.21.15.16:0/479137592 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+73 (secure 0 0 0) 0x7fbf80079610 con 0x7fbf8c0ce5f0 2024-04-24T23:50:28.543 INFO:teuthology.orchestra.run.smithi016.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T23:50:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.545+0000 7fbf922ba640 1 -- 172.21.15.16:0/479137592 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbf7806be60 msgr2=0x7fbf7806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.545+0000 7fbf922ba640 1 --2- 172.21.15.16:0/479137592 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbf7806be60 0x7fbf7806e320 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c000f60 tx=0x7fbf7c007d50 comp rx=0 tx=0).stop 2024-04-24T23:50:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.545+0000 7fbf922ba640 1 -- 172.21.15.16:0/479137592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf8c0ce5f0 msgr2=0x7fbf8c15a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.545+0000 7fbf922ba640 1 --2- 172.21.15.16:0/479137592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf8c0ce5f0 0x7fbf8c15a340 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fbf80035040 tx=0x7fbf800046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.545+0000 7fbf922ba640 1 -- 172.21.15.16:0/479137592 shutdown_connections 2024-04-24T23:50:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.545+0000 7fbf922ba640 1 --2- 172.21.15.16:0/479137592 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbf7806be60 0x7fbf7806e320 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.545+0000 7fbf922ba640 1 --2- 172.21.15.16:0/479137592 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbf8c15dfa0 0x7fbf8c155950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.545+0000 7fbf922ba640 1 --2- 172.21.15.16:0/479137592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbf8c0ce5f0 0x7fbf8c15a340 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.545+0000 7fbf922ba640 1 --2- 172.21.15.16:0/479137592 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbf8c0ccad0 0x7fbf8c152f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.545+0000 7fbf922ba640 1 -- 172.21.15.16:0/479137592 >> 172.21.15.16:0/479137592 conn(0x7fbf8c06b4d0 msgr2=0x7fbf8c0ccf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.545+0000 7fbf922ba640 1 -- 172.21.15.16:0/479137592 shutdown_connections 2024-04-24T23:50:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28.545+0000 7fbf922ba640 1 -- 172.21.15.16:0/479137592 wait complete. 2024-04-24T23:50:28.556 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T23:50:28.556 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T23:50:28.557 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi016.front.sepia.ceph.com 2024-04-24T23:50:28.557 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T23:50:28.557 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T23:50:28.557 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T23:50:28.557 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T23:50:28.557 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T23:50:28.557 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T23:50:28.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:28.558 DEBUG:teuthology.orchestra.run.smithi016:> ip addr 2024-04-24T23:50:28.574 INFO:teuthology.orchestra.run.smithi016.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T23:50:28.574 INFO:teuthology.orchestra.run.smithi016.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T23:50:28.574 INFO:teuthology.orchestra.run.smithi016.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T23:50:28.574 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:50:28.574 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 ::1/128 scope host 2024-04-24T23:50:28.574 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:50:28.574 INFO:teuthology.orchestra.run.smithi016.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:50:28.575 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e0 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:50:28.575 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f0 2024-04-24T23:50:28.575 INFO:teuthology.orchestra.run.smithi016.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T23:50:28.575 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:64 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:50:28.575 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f0 2024-04-24T23:50:28.575 INFO:teuthology.orchestra.run.smithi016.stdout: inet 172.21.15.16/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-04-24T23:50:28.575 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft 42647sec preferred_lft 42647sec 2024-04-24T23:50:28.575 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-04-24T23:50:28.575 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:50:28.575 INFO:teuthology.orchestra.run.smithi016.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:50:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:6b:fd:e1 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:50:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp7s0f1 2024-04-24T23:50:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T23:50:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 0c:c4:7a:bd:15:65 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:50:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: altname enp1s0f1 2024-04-24T23:50:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-04-24T23:50:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 16:ff:34:db:3f:c3 brd ff:ff:ff:ff:ff:ff 2024-04-24T23:50:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-04-24T23:50:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:50:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::303d:34ff:fea7:344f/64 scope link 2024-04-24T23:50:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:50:28.576 INFO:teuthology.orchestra.run.smithi016.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T23:50:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 82:fd:3d:cf:63:61 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:50:28.576 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::80fd:3dff:fecf:6361/64 scope link 2024-04-24T23:50:28.577 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:50:28.577 INFO:teuthology.orchestra.run.smithi016.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-04-24T23:50:28.577 INFO:teuthology.orchestra.run.smithi016.stdout: link/ether 16:ff:34:db:3f:c3 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T23:50:28.577 INFO:teuthology.orchestra.run.smithi016.stdout: inet6 fe80::14ff:34ff:fedb:3fc3/64 scope link 2024-04-24T23:50:28.577 INFO:teuthology.orchestra.run.smithi016.stdout: valid_lft forever preferred_lft forever 2024-04-24T23:50:28.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:28.578 DEBUG:teuthology.orchestra.run.smithi016:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T23:50:28.638 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-04-24T23:50:28.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:28.639 DEBUG:teuthology.orchestra.run.smithi016:> route 2024-04-24T23:50:28.696 INFO:teuthology.orchestra.run.smithi016.stdout:Kernel IP routing table 2024-04-24T23:50:28.696 INFO:teuthology.orchestra.run.smithi016.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T23:50:28.696 INFO:teuthology.orchestra.run.smithi016.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-24T23:50:28.696 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-24T23:50:28.696 INFO:teuthology.orchestra.run.smithi016.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T23:50:28.697 DEBUG:teuthology.orchestra.run.smithi016:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T23:50:28.697 DEBUG:teuthology.orchestra.run.smithi016:> set -e 2024-04-24T23:50:28.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-24T23:50:28.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-24T23:50:28.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-24T23:50:28.697 DEBUG:teuthology.orchestra.run.smithi016:> ') 2024-04-24T23:50:28.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T23:50:28.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:50:28.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:28.824 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-24T23:50:28.839 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T23:50:28.839 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T23:50:28.839 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T23:50:28.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:28.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link set brx.0 up 2024-04-24T23:50:28.904 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-04-24T23:50:28.905 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:50:28.905 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T23:50:28.959 INFO:teuthology.orchestra.run.smithi016.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T23:50:28.960 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T23:50:28.960 DEBUG:teuthology.orchestra.run.smithi016:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T23:50:29.014 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-24T23:50:29.079 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:50:29.135 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-24T23:50:29.135 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-24T23:50:29.136 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-24T23:50:29.137 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-24T23:50:29.137 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-04-24T23:50:29.137 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-24T23:50:29.137 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:50:29.137 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:50:29.137 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:50:29.137 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:50:29.137 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-24T23:50:29.137 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-24T23:50:29.137 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T23:50:29.137 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T23:50:29.137 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T23:50:29.137 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T23:50:29.137 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-24T23:50:29.138 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:50:29.138 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:50:29.138 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T23:50:29.138 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T23:50:29.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:29.138 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-24T23:50:29.191 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-24T23:50:29.191 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-04-24T23:50:29.234 DEBUG:teuthology.orchestra.run.smithi016:> sudo modprobe fuse 2024-04-24T23:50:29.258 DEBUG:teuthology.orchestra.run.smithi016:> cat /proc/self/mounts | awk '{print $2}' 2024-04-24T23:50:29.315 INFO:teuthology.orchestra.run.smithi016.stdout:/proc 2024-04-24T23:50:29.315 INFO:teuthology.orchestra.run.smithi016.stdout:/sys 2024-04-24T23:50:29.315 INFO:teuthology.orchestra.run.smithi016.stdout:/dev 2024-04-24T23:50:29.315 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/security 2024-04-24T23:50:29.315 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/shm 2024-04-24T23:50:29.315 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/pts 2024-04-24T23:50:29.315 INFO:teuthology.orchestra.run.smithi016.stdout:/run 2024-04-24T23:50:29.315 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/cgroup 2024-04-24T23:50:29.316 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/pstore 2024-04-24T23:50:29.316 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/bpf 2024-04-24T23:50:29.316 INFO:teuthology.orchestra.run.smithi016.stdout:/ 2024-04-24T23:50:29.316 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/selinux 2024-04-24T23:50:29.316 INFO:teuthology.orchestra.run.smithi016.stdout:/proc/sys/fs/binfmt_misc 2024-04-24T23:50:29.316 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/hugepages 2024-04-24T23:50:29.316 INFO:teuthology.orchestra.run.smithi016.stdout:/dev/mqueue 2024-04-24T23:50:29.316 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/debug 2024-04-24T23:50:29.316 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/tracing 2024-04-24T23:50:29.316 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/kernel/config 2024-04-24T23:50:29.316 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysusers.service 2024-04-24T23:50:29.317 INFO:teuthology.orchestra.run.smithi016.stdout:/sys/fs/fuse/connections 2024-04-24T23:50:29.317 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-sysctl.service 2024-04-24T23:50:29.317 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-24T23:50:29.317 INFO:teuthology.orchestra.run.smithi016.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-24T23:50:29.317 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/nfs/rpc_pipefs 2024-04-24T23:50:29.317 INFO:teuthology.orchestra.run.smithi016.stdout:/run/user/1000 2024-04-24T23:50:29.317 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph 2024-04-24T23:50:29.317 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-0 2024-04-24T23:50:29.317 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-1 2024-04-24T23:50:29.317 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-2 2024-04-24T23:50:29.318 INFO:teuthology.orchestra.run.smithi016.stdout:/var/lib/ceph/osd/ceph-3 2024-04-24T23:50:29.318 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns 2024-04-24T23:50:29.318 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:50:29.318 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:50:29.318 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T23:50:29.318 INFO:teuthology.orchestra.run.smithi016.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T23:50:29.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:29.318 DEBUG:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2024-04-24T23:50:29.319 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2024-04-24T23:50:29.316+0000 7fa100f09480 -1 init, newargv = 0x55e60b46fd80 newargc=15 2024-04-24T23:50:29.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[51121]: starting ceph client 2024-04-24T23:50:29.320 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[51121]: starting fuse 2024-04-24T23:50:29.334 INFO:teuthology.orchestra.run.smithi016.stdout:43 2024-04-24T23:50:29.334 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [43] 2024-04-24T23:50:29.335 DEBUG:teuthology.orchestra.run.smithi016:> sudo stdin-killer -- python3 -c ' 2024-04-24T23:50:29.335 DEBUG:teuthology.orchestra.run.smithi016:> import glob 2024-04-24T23:50:29.335 DEBUG:teuthology.orchestra.run.smithi016:> import re 2024-04-24T23:50:29.335 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-04-24T23:50:29.335 DEBUG:teuthology.orchestra.run.smithi016:> import subprocess 2024-04-24T23:50:29.335 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:50:29.335 DEBUG:teuthology.orchestra.run.smithi016:> def _find_admin_socket(client_name): 2024-04-24T23:50:29.335 DEBUG:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-04-24T23:50:29.335 DEBUG:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2024-04-24T23:50:29.335 DEBUG:teuthology.orchestra.run.smithi016:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-04-24T23:50:29.335 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:50:29.335 DEBUG:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2024-04-24T23:50:29.335 DEBUG:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2024-04-24T23:50:29.335 DEBUG:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2024-04-24T23:50:29.336 DEBUG:teuthology.orchestra.run.smithi016:> return files[0] 2024-04-24T23:50:29.336 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:50:29.336 DEBUG:teuthology.orchestra.run.smithi016:> for f in files: 2024-04-24T23:50:29.336 DEBUG:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-24T23:50:29.336 DEBUG:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-24T23:50:29.336 DEBUG:teuthology.orchestra.run.smithi016:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-24T23:50:29.336 DEBUG:teuthology.orchestra.run.smithi016:> contents = proc_f.read() 2024-04-24T23:50:29.336 DEBUG:teuthology.orchestra.run.smithi016:> if mountpoint in contents: 2024-04-24T23:50:29.336 DEBUG:teuthology.orchestra.run.smithi016:> return f 2024-04-24T23:50:29.336 DEBUG:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-24T23:50:29.336 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:50:29.336 DEBUG:teuthology.orchestra.run.smithi016:> print(_find_admin_socket("client.0")) 2024-04-24T23:50:29.336 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-04-24T23:50:29.453 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.51121.asok 2024-04-24T23:50:29.456 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:50:29.464 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.51121.asok 2024-04-24T23:50:29.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:29.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.51121.asok status 2024-04-24T23:50:29.599 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:50:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2024-04-24T23:50:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "d2c45d977142e024a59b68a8f1dce8f0b9840551", 2024-04-24T23:50:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-2445-gd2c45d97 (d2c45d977142e024a59b68a8f1dce8f0b9840551) squid (dev)", 2024-04-24T23:50:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2024-04-24T23:50:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2024-04-24T23:50:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-04-24T23:50:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "51121", 2024-04-24T23:50:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2024-04-24T23:50:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2024-04-24T23:50:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2024-04-24T23:50:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 5724, 2024-04-24T23:50:29.600 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2024-04-24T23:50:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2024-04-24T23:50:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2024-04-24T23:50:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 5724 2024-04-24T23:50:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-04-24T23:50:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-24T23:50:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-04-24T23:50:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3126049350 2024-04-24T23:50:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: } 2024-04-24T23:50:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2024-04-24T23:50:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2024-04-24T23:50:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "192.168.0.1:0", 2024-04-24T23:50:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 3126049350 2024-04-24T23:50:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2024-04-24T23:50:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.5724 192.168.0.1:0/3126049350", 2024-04-24T23:50:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "192.168.0.1:0/3126049350", 2024-04-24T23:50:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2024-04-24T23:50:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 15, 2024-04-24T23:50:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 39, 2024-04-24T23:50:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2024-04-24T23:50:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "blocklisted": false, 2024-04-24T23:50:29.602 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs" 2024-04-24T23:50:29.602 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:50:29.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:29.612 DEBUG:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2024-04-24T23:50:29.613 DEBUG:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2024-04-24T23:50:29.628 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2024-04-24T23:50:29.628 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-04-24T23:50:29.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:29.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T23:50:29.693 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:50:29.694 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:50:29.749 DEBUG:teuthology.orchestra.run.smithi016:> stdin-killer -- python3 -c ' 2024-04-24T23:50:29.749 DEBUG:teuthology.orchestra.run.smithi016:> import os 2024-04-24T23:50:29.749 DEBUG:teuthology.orchestra.run.smithi016:> import uuid 2024-04-24T23:50:29.749 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:50:29.749 DEBUG:teuthology.orchestra.run.smithi016:> n = 1 2024-04-24T23:50:29.749 DEBUG:teuthology.orchestra.run.smithi016:> create_hard_links = False 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> if 0 > 0: 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> create_hard_links = True 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> path = "/home/ubuntu/cephtest/mnt.0/file_on_fs" 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> dpath = os.path.dirname(path) 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> fnameprefix = os.path.basename(path) 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> os.makedirs(dpath, exist_ok=True) 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> try: 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> for i in range(n): 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> with open(fpath, '"'"'w'"'"') as f: 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> f.write(f"{i}") 2024-04-24T23:50:29.750 DEBUG:teuthology.orchestra.run.smithi016:> if True: 2024-04-24T23:50:29.751 DEBUG:teuthology.orchestra.run.smithi016:> f.flush() 2024-04-24T23:50:29.751 DEBUG:teuthology.orchestra.run.smithi016:> os.fsync(f.fileno()) 2024-04-24T23:50:29.751 DEBUG:teuthology.orchestra.run.smithi016:> if False: 2024-04-24T23:50:29.751 DEBUG:teuthology.orchestra.run.smithi016:> os.unlink(fpath) 2024-04-24T23:50:29.751 DEBUG:teuthology.orchestra.run.smithi016:> if False: 2024-04-24T23:50:29.751 DEBUG:teuthology.orchestra.run.smithi016:> os.fsync(dirfd) 2024-04-24T23:50:29.751 DEBUG:teuthology.orchestra.run.smithi016:> if create_hard_links: 2024-04-24T23:50:29.751 DEBUG:teuthology.orchestra.run.smithi016:> for j in range(0): 2024-04-24T23:50:29.751 DEBUG:teuthology.orchestra.run.smithi016:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-04-24T23:50:29.751 DEBUG:teuthology.orchestra.run.smithi016:> if False: 2024-04-24T23:50:29.751 DEBUG:teuthology.orchestra.run.smithi016:> os.fsync(dirfd) 2024-04-24T23:50:29.751 DEBUG:teuthology.orchestra.run.smithi016:> finally: 2024-04-24T23:50:29.751 DEBUG:teuthology.orchestra.run.smithi016:> os.close(dirfd) 2024-04-24T23:50:29.751 DEBUG:teuthology.orchestra.run.smithi016:> ' 2024-04-24T23:50:29.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T23:50:29.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T23:50:30.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.024+0000 7f781ccee640 1 Processor -- start 2024-04-24T23:50:30.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.024+0000 7f781ccee640 1 -- start start 2024-04-24T23:50:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.025+0000 7f781ccee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781806cb00 0x7f781815c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.025+0000 7f781ccee640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78180ccc70 0x7f7818072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.025+0000 7f781ccee640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78180cfd50 con 0x7f78180727d0 2024-04-24T23:50:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.025+0000 7f781ccee640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f78180ce760 con 0x7f78180ccc70 2024-04-24T23:50:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.025+0000 7f781ccee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f781806cee0 con 0x7f781806cb00 2024-04-24T23:50:30.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.025+0000 7f7815d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78180ccc70 0x7f7818072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:30.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.025+0000 7f7815d9b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f78180ccc70 0x7f7818072290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38126/0 (socket says 172.21.15.16:38126) 2024-04-24T23:50:30.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.025+0000 7f7815d9b640 1 -- 172.21.15.16:0/3517889636 learned_addr learned my addr 172.21.15.16:0/3517889636 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:30.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.025+0000 7f781659c640 1 --2- 172.21.15.16:0/3517889636 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781806cb00 0x7f781815c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:30.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.025+0000 7f7815d9b640 1 -- 172.21.15.16:0/3517889636 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781806cb00 msgr2=0x7f781815c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:30.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.025+0000 7f7815d9b640 1 --2- 172.21.15.16:0/3517889636 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781806cb00 0x7f781815c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.025+0000 7f7815d9b640 1 -- 172.21.15.16:0/3517889636 >> v1:172.21.15.156:6789/0 conn(0x7f78180727d0 legacy=0x7f781815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:30.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.025+0000 7f7815d9b640 1 -- 172.21.15.16:0/3517889636 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f781815c8b0 con 0x7f78180ccc70 2024-04-24T23:50:30.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.026+0000 7f7815d9b640 1 --2- 172.21.15.16:0/3517889636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180ccc70 0x7f7818072290 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7818067d50 tx=0x7f780c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48a060958fc9bccf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:30.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.026+0000 7f781559a640 1 -- 172.21.15.16:0/3517889636 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c037070 con 0x7f78180ccc70 2024-04-24T23:50:30.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.026+0000 7f781559a640 1 -- 172.21.15.16:0/3517889636 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f780c032440 con 0x7f78180ccc70 2024-04-24T23:50:30.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.026+0000 7f781559a640 1 -- 172.21.15.16:0/3517889636 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c032810 con 0x7f78180ccc70 2024-04-24T23:50:30.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.027+0000 7f781ccee640 1 -- 172.21.15.16:0/3517889636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180ccc70 msgr2=0x7f7818072290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:30.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.027+0000 7f781ccee640 1 --2- 172.21.15.16:0/3517889636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180ccc70 0x7f7818072290 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f7818067d50 tx=0x7f780c02a1e0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.027+0000 7f781ccee640 1 -- 172.21.15.16:0/3517889636 shutdown_connections 2024-04-24T23:50:30.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.027+0000 7f781ccee640 1 --2- 172.21.15.16:0/3517889636 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180ccc70 0x7f7818072290 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.027+0000 7f781ccee640 1 --2- 172.21.15.16:0/3517889636 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781806cb00 0x7f781815c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.027+0000 7f781ccee640 1 -- 172.21.15.16:0/3517889636 >> 172.21.15.16:0/3517889636 conn(0x7f78180d0260 msgr2=0x7f78180d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:30.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.027+0000 7f781ccee640 1 -- 172.21.15.16:0/3517889636 shutdown_connections 2024-04-24T23:50:30.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.027+0000 7f781ccee640 1 -- 172.21.15.16:0/3517889636 wait complete. 2024-04-24T23:50:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.028+0000 7f781ccee640 1 Processor -- start 2024-04-24T23:50:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.028+0000 7f781ccee640 1 -- start start 2024-04-24T23:50:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f781ccee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781806cb00 0x7f78180aef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f781ccee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180727d0 0x7f781815b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f781ccee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78180ccc70 0x7f781815b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f781ccee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f781815d150 con 0x7f78180727d0 2024-04-24T23:50:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f781ccee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f781815ce70 con 0x7f78180ccc70 2024-04-24T23:50:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f781ccee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f781806b0e0 con 0x7f781806cb00 2024-04-24T23:50:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f7815d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180727d0 0x7f781815b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:30.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f7815d9b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180727d0 0x7f781815b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38142/0 (socket says 172.21.15.16:38142) 2024-04-24T23:50:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f7815d9b640 1 -- 172.21.15.16:0/3317195698 learned_addr learned my addr 172.21.15.16:0/3317195698 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f781659c640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781806cb00 0x7f78180aef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f7816d9d640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78180ccc70 0x7f781815b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f7815d9b640 1 -- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781806cb00 msgr2=0x7f78180aef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f7815d9b640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781806cb00 0x7f78180aef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f7815d9b640 1 -- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78180ccc70 msgr2=0x7f781815b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f7815d9b640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78180ccc70 0x7f781815b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f7815d9b640 1 -- 172.21.15.16:0/3317195698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7818142be0 con 0x7f78180727d0 2024-04-24T23:50:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f781659c640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781806cb00 0x7f78180aef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.029+0000 7f7816d9d640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78180ccc70 0x7f781815b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.030+0000 7f7815d9b640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180727d0 0x7f781815b210 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f780c000c00 tx=0x7f780c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.030+0000 7f7806ffd640 1 -- 172.21.15.16:0/3317195698 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c041020 con 0x7f78180727d0 2024-04-24T23:50:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.030+0000 7f7806ffd640 1 -- 172.21.15.16:0/3317195698 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f780c0306c0 con 0x7f78180727d0 2024-04-24T23:50:30.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.030+0000 7f781ccee640 1 -- 172.21.15.16:0/3317195698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7818142ec0 con 0x7f78180727d0 2024-04-24T23:50:30.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.030+0000 7f7806ffd640 1 -- 172.21.15.16:0/3317195698 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f780c037040 con 0x7f78180727d0 2024-04-24T23:50:30.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.030+0000 7f781ccee640 1 -- 172.21.15.16:0/3317195698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7818158410 con 0x7f78180727d0 2024-04-24T23:50:30.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.031+0000 7f781ccee640 1 -- 172.21.15.16:0/3317195698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f781804f5e0 con 0x7f78180727d0 2024-04-24T23:50:30.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.032+0000 7f7806ffd640 1 -- 172.21.15.16:0/3317195698 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f780c044020 con 0x7f78180727d0 2024-04-24T23:50:30.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.032+0000 7f7806ffd640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f77f806be30 0x7f77f806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:30.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.032+0000 7f7806ffd640 1 -- 172.21.15.16:0/3317195698 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 6101+0+0 (secure 0 0 0) 0x7f780c0ab830 con 0x7f78180727d0 2024-04-24T23:50:30.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.032+0000 7f781659c640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f77f806be30 0x7f77f806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:30.033 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.033+0000 7f781659c640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f77f806be30 0x7f77f806e2f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7810006fd0 tx=0x7f7810008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:30.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.036+0000 7f7806ffd640 1 -- 172.21.15.16:0/3317195698 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f780c04b030 con 0x7f78180727d0 2024-04-24T23:50:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.258+0000 7f781ccee640 1 -- 172.21.15.16:0/3317195698 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f7818072e90 con 0x7f78180727d0 2024-04-24T23:50:30.258 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-04-24T23:50:30.259+0000 7fa77c6c1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T23:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.260+0000 7f7806ffd640 1 -- 172.21.15.16:0/3317195698 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) ==== 161+0+0 (secure 0 0 0) 0x7f780c07a050 con 0x7f78180727d0 2024-04-24T23:50:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T23:50:30.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.263+0000 7f781ccee640 1 -- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f77f806be30 msgr2=0x7f77f806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:30.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.263+0000 7f781ccee640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f77f806be30 0x7f77f806e2f0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7810006fd0 tx=0x7f7810008040 comp rx=0 tx=0).stop 2024-04-24T23:50:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.263+0000 7f781ccee640 1 -- 172.21.15.16:0/3317195698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180727d0 msgr2=0x7f781815b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.263+0000 7f781ccee640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180727d0 0x7f781815b210 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f780c000c00 tx=0x7f780c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.263+0000 7f781ccee640 1 -- 172.21.15.16:0/3317195698 shutdown_connections 2024-04-24T23:50:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.263+0000 7f781ccee640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f77f806be30 0x7f77f806e2f0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.263+0000 7f781ccee640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f78180ccc70 0x7f781815b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.263+0000 7f781ccee640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180727d0 0x7f781815b210 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.263+0000 7f781ccee640 1 --2- 172.21.15.16:0/3317195698 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f781806cb00 0x7f78180aef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.263+0000 7f781ccee640 1 -- 172.21.15.16:0/3317195698 >> 172.21.15.16:0/3317195698 conn(0x7f78180d0260 msgr2=0x7f78180b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.264+0000 7f781ccee640 1 -- 172.21.15.16:0/3317195698 shutdown_connections 2024-04-24T23:50:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.264+0000 7f781ccee640 1 -- 172.21.15.16:0/3317195698 wait complete. 2024-04-24T23:50:30.265 INFO:tasks.ceph.mds.a.smithi016.stderr: -1027> 2024-04-24T23:50:23.052+0000 7f439e3a5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T23:50:30.274 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T23:50:30.290 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-04-24T23:50:30.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.378+0000 7ff59502c640 1 Processor -- start 2024-04-24T23:50:30.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.379+0000 7ff59502c640 1 -- start start 2024-04-24T23:50:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.379+0000 7ff59502c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff59015dfd0 0x7ff59006b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.379+0000 7ff59502c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5900cfde0 con 0x7ff59006bc40 2024-04-24T23:50:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.379+0000 7ff59502c640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5900d24d0 con 0x7ff5900b8060 2024-04-24T23:50:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.379+0000 7ff59502c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ff5900d27b0 con 0x7ff59015dfd0 2024-04-24T23:50:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.379+0000 7ff58e59c640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff59006bc40 0x7ff5900cefc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55924/0 (socket says 172.21.15.16:55924) 2024-04-24T23:50:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.379+0000 7ff58e59c640 1 -- 172.21.15.16:0/2825670655 learned_addr learned my addr 172.21.15.16:0/2825670655 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.380+0000 7ff58ed9d640 1 --2- 172.21.15.16:0/2825670655 >> v2:172.21.15.156:3300/0 conn(0x7ff59015dfd0 0x7ff59006b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.380+0000 7ff58dd9b640 1 -- 172.21.15.16:0/2825670655 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 94601685 0 0) 0x7ff5900cfde0 con 0x7ff59006bc40 2024-04-24T23:50:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.380+0000 7ff58dd9b640 1 -- 172.21.15.16:0/2825670655 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff57c003520 con 0x7ff59006bc40 2024-04-24T23:50:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.380+0000 7ff58ed9d640 1 -- 172.21.15.16:0/2825670655 >> v1:172.21.15.156:6789/0 conn(0x7ff5900b8060 legacy=0x7ff5900cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.380+0000 7ff58dd9b640 1 -- 172.21.15.16:0/2825670655 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4197740820 0 0) 0x7ff57c003520 con 0x7ff59006bc40 2024-04-24T23:50:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.380+0000 7ff58ed9d640 1 -- 172.21.15.16:0/2825670655 >> v1:172.21.15.16:6789/0 conn(0x7ff59006bc40 legacy=0x7ff5900cefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:30.380 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.380+0000 7ff58ed9d640 1 -- 172.21.15.16:0/2825670655 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5900d2a90 con 0x7ff59015dfd0 2024-04-24T23:50:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.381+0000 7ff58ed9d640 1 --2- 172.21.15.16:0/2825670655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff59015dfd0 0x7ff59006b2f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff580009d90 tx=0x7ff58002a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e013b8165cea3057 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.381+0000 7ff58dd9b640 1 -- 172.21.15.16:0/2825670655 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff578001670 con 0x7ff59015dfd0 2024-04-24T23:50:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.381+0000 7ff58dd9b640 1 -- 172.21.15.16:0/2825670655 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff580007540 con 0x7ff59015dfd0 2024-04-24T23:50:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.381+0000 7ff58dd9b640 1 -- 172.21.15.16:0/2825670655 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580007910 con 0x7ff59015dfd0 2024-04-24T23:50:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.381+0000 7ff59502c640 1 -- 172.21.15.16:0/2825670655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff59015dfd0 msgr2=0x7ff59006b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.381+0000 7ff59502c640 1 --2- 172.21.15.16:0/2825670655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff59015dfd0 0x7ff59006b2f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff580009d90 tx=0x7ff58002a5f0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.381+0000 7ff59502c640 1 -- 172.21.15.16:0/2825670655 shutdown_connections 2024-04-24T23:50:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.381+0000 7ff59502c640 1 --2- 172.21.15.16:0/2825670655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff59015dfd0 0x7ff59006b2f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.381+0000 7ff59502c640 1 -- 172.21.15.16:0/2825670655 >> 172.21.15.16:0/2825670655 conn(0x7ff5900d3dc0 msgr2=0x7ff5900d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.381+0000 7ff59502c640 1 -- 172.21.15.16:0/2825670655 shutdown_connections 2024-04-24T23:50:30.382 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.382+0000 7ff59502c640 1 -- 172.21.15.16:0/2825670655 wait complete. 2024-04-24T23:50:30.384 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.383+0000 7ff59502c640 1 Processor -- start 2024-04-24T23:50:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.383+0000 7ff59502c640 1 -- start start 2024-04-24T23:50:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff59502c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59006bc40 0x7ff5901f3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff59502c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5900b8060 0x7ff590159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff59502c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff59015dfd0 0x7ff59015a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff59502c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff5900cead0 con 0x7ff59006bc40 2024-04-24T23:50:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff59502c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff5900d3050 con 0x7ff5900b8060 2024-04-24T23:50:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff59502c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff59006b830 con 0x7ff59015dfd0 2024-04-24T23:50:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff58ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59006bc40 0x7ff5901f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff58ed9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59006bc40 0x7ff5901f3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38170/0 (socket says 172.21.15.16:38170) 2024-04-24T23:50:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff58ed9d640 1 -- 172.21.15.16:0/284052732 learned_addr learned my addr 172.21.15.16:0/284052732 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff58f59e640 1 --2- 172.21.15.16:0/284052732 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff59015dfd0 0x7ff59015a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff58e59c640 1 --2- 172.21.15.16:0/284052732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5900b8060 0x7ff590159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:30.385 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff58ed9d640 1 -- 172.21.15.16:0/284052732 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff59015dfd0 msgr2=0x7ff59015a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff58ed9d640 1 --2- 172.21.15.16:0/284052732 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff59015dfd0 0x7ff59015a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff58ed9d640 1 -- 172.21.15.16:0/284052732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5900b8060 msgr2=0x7ff590159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff58ed9d640 1 --2- 172.21.15.16:0/284052732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5900b8060 0x7ff590159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff58ed9d640 1 -- 172.21.15.16:0/284052732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59015a140 con 0x7ff59006bc40 2024-04-24T23:50:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff58ed9d640 1 --2- 172.21.15.16:0/284052732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59006bc40 0x7ff5901f3430 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff580009d90 tx=0x7ff580004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff5877fe640 1 -- 172.21.15.16:0/284052732 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580040070 con 0x7ff59006bc40 2024-04-24T23:50:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff5877fe640 1 -- 172.21.15.16:0/284052732 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff580031bc0 con 0x7ff59006bc40 2024-04-24T23:50:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff59502c640 1 -- 172.21.15.16:0/284052732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff59015e550 con 0x7ff59006bc40 2024-04-24T23:50:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.384+0000 7ff5877fe640 1 -- 172.21.15.16:0/284052732 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580009450 con 0x7ff59006bc40 2024-04-24T23:50:30.386 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.385+0000 7ff59502c640 1 -- 172.21.15.16:0/284052732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff590160990 con 0x7ff59006bc40 2024-04-24T23:50:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.385+0000 7ff59502c640 1 -- 172.21.15.16:0/284052732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff590068f90 con 0x7ff59006bc40 2024-04-24T23:50:30.387 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.386+0000 7ff5877fe640 1 -- 172.21.15.16:0/284052732 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff580039040 con 0x7ff59006bc40 2024-04-24T23:50:30.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.386+0000 7ff5877fe640 1 --2- 172.21.15.16:0/284052732 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff57006be30 0x7ff57006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:30.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.386+0000 7ff5877fe640 1 -- 172.21.15.16:0/284052732 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6187+0+0 (secure 0 0 0) 0x7ff5800ac860 con 0x7ff59006bc40 2024-04-24T23:50:30.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.387+0000 7ff58e59c640 1 --2- 172.21.15.16:0/284052732 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff57006be30 0x7ff57006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:30.388 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.387+0000 7ff58e59c640 1 --2- 172.21.15.16:0/284052732 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff57006be30 0x7ff57006e2f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff5901f6f70 tx=0x7ff578009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:30.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.390+0000 7ff5877fe640 1 -- 172.21.15.16:0/284052732 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff5800443c0 con 0x7ff59006bc40 2024-04-24T23:50:30.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:30.613+0000 7ff59502c640 1 -- 172.21.15.16:0/284052732 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff5900b5a50 con 0x7ff59006bc40 2024-04-24T23:50:31.263 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.264+0000 7ff5877fe640 1 -- 172.21.15.16:0/284052732 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v17) ==== 104+0+0 (secure 0 0 0) 0x7ff58007b030 con 0x7ff59006bc40 2024-04-24T23:50:31.266 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.267+0000 7ff59502c640 1 -- 172.21.15.16:0/284052732 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff57006be30 msgr2=0x7ff57006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.267+0000 7ff59502c640 1 --2- 172.21.15.16:0/284052732 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff57006be30 0x7ff57006e2f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff5901f6f70 tx=0x7ff578009040 comp rx=0 tx=0).stop 2024-04-24T23:50:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.267+0000 7ff59502c640 1 -- 172.21.15.16:0/284052732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59006bc40 msgr2=0x7ff5901f3430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.267+0000 7ff59502c640 1 --2- 172.21.15.16:0/284052732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59006bc40 0x7ff5901f3430 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ff580009d90 tx=0x7ff580004a40 comp rx=0 tx=0).stop 2024-04-24T23:50:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.267+0000 7ff59502c640 1 -- 172.21.15.16:0/284052732 shutdown_connections 2024-04-24T23:50:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.267+0000 7ff59502c640 1 --2- 172.21.15.16:0/284052732 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff57006be30 0x7ff57006e2f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.267+0000 7ff59502c640 1 --2- 172.21.15.16:0/284052732 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff59015dfd0 0x7ff59015a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.267+0000 7ff59502c640 1 --2- 172.21.15.16:0/284052732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5900b8060 0x7ff590159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.267+0000 7ff59502c640 1 --2- 172.21.15.16:0/284052732 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff59006bc40 0x7ff5901f3430 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.267+0000 7ff59502c640 1 -- 172.21.15.16:0/284052732 >> 172.21.15.16:0/284052732 conn(0x7ff5900d3dc0 msgr2=0x7ff5900b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.267+0000 7ff59502c640 1 -- 172.21.15.16:0/284052732 shutdown_connections 2024-04-24T23:50:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.267+0000 7ff59502c640 1 -- 172.21.15.16:0/284052732 wait complete. 2024-04-24T23:50:31.281 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --recover --force --fscid 2 2024-04-24T23:50:31.437 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.438+0000 7f6b73bb7640 1 Processor -- start 2024-04-24T23:50:31.438 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.438+0000 7f6b73bb7640 1 -- start start 2024-04-24T23:50:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b73bb7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6b6c15dff0 0x7f6b6c06deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b73bb7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b6c06d8d0 con 0x7f6b6c0af980 2024-04-24T23:50:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b73bb7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6b6c06dbb0 con 0x7f6b6c06d360 2024-04-24T23:50:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b73bb7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f6b6c0732d0 con 0x7f6b6c15dff0 2024-04-24T23:50:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b72154640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6b6c0af980 0x7f6b6c071e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55930/0 (socket says 172.21.15.16:55930) 2024-04-24T23:50:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b72154640 1 -- 172.21.15.16:0/3922667166 learned_addr learned my addr 172.21.15.16:0/3922667166 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b70951640 1 -- 172.21.15.16:0/3922667166 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 20583789 0 0) 0x7f6b6c06d8d0 con 0x7f6b6c0af980 2024-04-24T23:50:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b71953640 1 --2- 172.21.15.16:0/3922667166 >> v2:172.21.15.156:3300/0 conn(0x7f6b6c15dff0 0x7f6b6c06deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b70951640 1 -- 172.21.15.16:0/3922667166 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6b58003520 con 0x7f6b6c0af980 2024-04-24T23:50:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b70951640 1 -- 172.21.15.16:0/3922667166 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2982989063 0 0) 0x7f6b58003520 con 0x7f6b6c0af980 2024-04-24T23:50:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b70951640 1 -- 172.21.15.16:0/3922667166 >> v2:172.21.15.156:3300/0 conn(0x7f6b6c15dff0 msgr2=0x7f6b6c06deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b70951640 1 --2- 172.21.15.16:0/3922667166 >> v2:172.21.15.156:3300/0 conn(0x7f6b6c15dff0 0x7f6b6c06deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b70951640 1 -- 172.21.15.16:0/3922667166 >> v1:172.21.15.156:6789/0 conn(0x7f6b6c06d360 legacy=0x7f6b6c0af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b70951640 1 -- 172.21.15.16:0/3922667166 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6c071330 con 0x7f6b6c0af980 2024-04-24T23:50:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b70951640 1 -- 172.21.15.16:0/3922667166 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f6b640019f0 con 0x7f6b6c0af980 2024-04-24T23:50:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.439+0000 7f6b71953640 1 --2- 172.21.15.16:0/3922667166 >> v2:172.21.15.156:3300/0 conn(0x7f6b6c15dff0 0x7f6b6c06deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b70951640 1 -- 172.21.15.16:0/3922667166 >> v1:172.21.15.16:6789/0 conn(0x7f6b6c0af980 legacy=0x7f6b6c071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b70951640 1 --2- 172.21.15.16:0/3922667166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b580039b0 0x7f6b58003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b70951640 1 --2- 172.21.15.16:0/3922667166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b58004800 0x7f6b58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b70951640 1 --2- 172.21.15.16:0/3922667166 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b580051c0 0x7f6b5800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b70951640 1 -- 172.21.15.16:0/3922667166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c0732d0 con 0x7f6b580039b0 2024-04-24T23:50:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b70951640 1 -- 172.21.15.16:0/3922667166 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6b5800c110 con 0x7f6b58004800 2024-04-24T23:50:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b70951640 1 -- 172.21.15.16:0/3922667166 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6b5800c310 con 0x7f6b580051c0 2024-04-24T23:50:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b71953640 1 --2- 172.21.15.16:0/3922667166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b580039b0 0x7f6b58003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b71152640 1 --2- 172.21.15.16:0/3922667166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b58004800 0x7f6b58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b72154640 1 --2- 172.21.15.16:0/3922667166 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b580051c0 0x7f6b5800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b71953640 1 -- 172.21.15.16:0/3922667166 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b580051c0 msgr2=0x7f6b5800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b71953640 1 --2- 172.21.15.16:0/3922667166 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b580051c0 0x7f6b5800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b71953640 1 -- 172.21.15.16:0/3922667166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b58004800 msgr2=0x7f6b58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b71953640 1 --2- 172.21.15.16:0/3922667166 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b58004800 0x7f6b58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b71953640 1 -- 172.21.15.16:0/3922667166 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b58003520 con 0x7f6b580039b0 2024-04-24T23:50:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.440+0000 7f6b72154640 1 -- 172.21.15.16:0/3922667166 reap_dead start 2024-04-24T23:50:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.441+0000 7f6b71953640 1 --2- 172.21.15.16:0/3922667166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b580039b0 0x7f6b58003dd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6b680098a0 tx=0x7f6b58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef235f58ba07a083 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.441+0000 7f6b70951640 1 -- 172.21.15.16:0/3922667166 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6800f040 con 0x7f6b580039b0 2024-04-24T23:50:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.441+0000 7f6b70951640 1 -- 172.21.15.16:0/3922667166 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b6802ace0 con 0x7f6b580039b0 2024-04-24T23:50:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.441+0000 7f6b70951640 1 -- 172.21.15.16:0/3922667166 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b68004250 con 0x7f6b580039b0 2024-04-24T23:50:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.441+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3922667166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b580039b0 msgr2=0x7f6b58003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.441+0000 7f6b73bb7640 1 --2- 172.21.15.16:0/3922667166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b580039b0 0x7f6b58003dd0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6b680098a0 tx=0x7f6b58005830 comp rx=0 tx=0).stop 2024-04-24T23:50:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.441+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3922667166 shutdown_connections 2024-04-24T23:50:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.441+0000 7f6b73bb7640 1 --2- 172.21.15.16:0/3922667166 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b580039b0 0x7f6b58003dd0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.441+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3922667166 >> 172.21.15.16:0/3922667166 conn(0x7f6b6c0affc0 msgr2=0x7f6b6c0b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.441+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3922667166 shutdown_connections 2024-04-24T23:50:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.441+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3922667166 wait complete. 2024-04-24T23:50:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.443+0000 7f6b73bb7640 1 Processor -- start 2024-04-24T23:50:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.443+0000 7f6b73bb7640 1 -- start start 2024-04-24T23:50:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.443+0000 7f6b73bb7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b580039b0 0x7f6b6c15f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.443+0000 7f6b73bb7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b6c155d20 0x7f6b6c148d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.443+0000 7f6b73bb7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b6c0ce050 0x7f6b6c0ce4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.443+0000 7f6b73bb7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c0ccca0 con 0x7f6b580039b0 2024-04-24T23:50:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.443+0000 7f6b73bb7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6b6c0718f0 con 0x7f6b6c0ce050 2024-04-24T23:50:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.443+0000 7f6b73bb7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6b6c0cce20 con 0x7f6b6c155d20 2024-04-24T23:50:31.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.443+0000 7f6b71953640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b580039b0 0x7f6b6c15f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b71953640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b580039b0 0x7f6b6c15f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38198/0 (socket says 172.21.15.16:38198) 2024-04-24T23:50:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b71953640 1 -- 172.21.15.16:0/3128461491 learned_addr learned my addr 172.21.15.16:0/3128461491 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b72154640 1 --2- 172.21.15.16:0/3128461491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b6c0ce050 0x7f6b6c0ce4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b71152640 1 --2- 172.21.15.16:0/3128461491 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b6c155d20 0x7f6b6c148d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b71953640 1 -- 172.21.15.16:0/3128461491 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b6c155d20 msgr2=0x7f6b6c148d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b71953640 1 --2- 172.21.15.16:0/3128461491 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b6c155d20 0x7f6b6c148d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b71953640 1 -- 172.21.15.16:0/3128461491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b6c0ce050 msgr2=0x7f6b6c0ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b71953640 1 --2- 172.21.15.16:0/3128461491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b6c0ce050 0x7f6b6c0ce4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b71953640 1 -- 172.21.15.16:0/3128461491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b6c1561a0 con 0x7f6b580039b0 2024-04-24T23:50:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b71953640 1 --2- 172.21.15.16:0/3128461491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b580039b0 0x7f6b6c15f9d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6b680098a0 tx=0x7f6b68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b627fc640 1 -- 172.21.15.16:0/3128461491 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b68041070 con 0x7f6b580039b0 2024-04-24T23:50:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b627fc640 1 -- 172.21.15.16:0/3128461491 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b6802aae0 con 0x7f6b580039b0 2024-04-24T23:50:31.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b627fc640 1 -- 172.21.15.16:0/3128461491 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b6803d430 con 0x7f6b580039b0 2024-04-24T23:50:31.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3128461491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b6c1600d0 con 0x7f6b580039b0 2024-04-24T23:50:31.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.444+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3128461491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b6c160690 con 0x7f6b580039b0 2024-04-24T23:50:31.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.445+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3128461491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b6c068f90 con 0x7f6b580039b0 2024-04-24T23:50:31.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.446+0000 7f6b627fc640 1 -- 172.21.15.16:0/3128461491 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6b68044070 con 0x7f6b580039b0 2024-04-24T23:50:31.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.446+0000 7f6b627fc640 1 --2- 172.21.15.16:0/3128461491 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6b5006be60 0x7f6b5006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:31.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.446+0000 7f6b627fc640 1 -- 172.21.15.16:0/3128461491 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 6187+0+0 (secure 0 0 0) 0x7f6b680ab6c0 con 0x7f6b580039b0 2024-04-24T23:50:31.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.447+0000 7f6b71152640 1 --2- 172.21.15.16:0/3128461491 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6b5006be60 0x7f6b5006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:31.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.447+0000 7f6b71152640 1 --2- 172.21.15.16:0/3128461491 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6b5006be60 0x7f6b5006e320 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c007fd0 tx=0x7f6b5c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:31.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.451+0000 7f6b627fc640 1 -- 172.21.15.16:0/3128461491 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6b680797b0 con 0x7f6b580039b0 2024-04-24T23:50:31.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:31.670+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3128461491 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "recover": true, "force": true, "fscid": 2} v 0) -- 0x7f6b6c161730 con 0x7f6b580039b0 2024-04-24T23:50:32.263 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-04-24T23:50:32.264+0000 7fa77eec6640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T23:50:32.265 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.265+0000 7f6b627fc640 1 -- 172.21.15.16:0/3128461491 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "recover": true, "force": true, "fscid": 2}]=0 new fs with metadata pool 5 and data pool 6 v18) ==== 216+0+0 (secure 0 0 0) 0x7f6b6807dab0 con 0x7f6b580039b0 2024-04-24T23:50:32.265 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 5 and data pool 6 2024-04-24T23:50:32.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.268+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3128461491 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6b5006be60 msgr2=0x7f6b5006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:32.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.268+0000 7f6b73bb7640 1 --2- 172.21.15.16:0/3128461491 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6b5006be60 0x7f6b5006e320 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c007fd0 tx=0x7f6b5c009040 comp rx=0 tx=0).stop 2024-04-24T23:50:32.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.268+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3128461491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b580039b0 msgr2=0x7f6b6c15f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:32.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.268+0000 7f6b73bb7640 1 --2- 172.21.15.16:0/3128461491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b580039b0 0x7f6b6c15f9d0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6b680098a0 tx=0x7f6b68004730 comp rx=0 tx=0).stop 2024-04-24T23:50:32.268 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.268+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3128461491 shutdown_connections 2024-04-24T23:50:32.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.268+0000 7f6b73bb7640 1 --2- 172.21.15.16:0/3128461491 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6b5006be60 0x7f6b5006e320 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:32.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.268+0000 7f6b73bb7640 1 --2- 172.21.15.16:0/3128461491 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6b6c0ce050 0x7f6b6c0ce4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:32.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.268+0000 7f6b73bb7640 1 --2- 172.21.15.16:0/3128461491 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6b6c155d20 0x7f6b6c148d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:32.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.268+0000 7f6b73bb7640 1 --2- 172.21.15.16:0/3128461491 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b580039b0 0x7f6b6c15f9d0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:32.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.268+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3128461491 >> 172.21.15.16:0/3128461491 conn(0x7f6b6c0affc0 msgr2=0x7f6b6c0b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:32.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.268+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3128461491 shutdown_connections 2024-04-24T23:50:32.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.268+0000 7f6b73bb7640 1 -- 172.21.15.16:0/3128461491 wait complete. 2024-04-24T23:50:32.280 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-04-24T23:50:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.390+0000 7f1b2d7f4640 1 Processor -- start 2024-04-24T23:50:32.390 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.390+0000 7f1b2d7f4640 1 -- start start 2024-04-24T23:50:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.391+0000 7f1b2d7f4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1b2815e2f0 0x7f1b2806b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.391+0000 7f1b2d7f4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1b280d26e0 0x7f1b2806b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.391+0000 7f1b2d7f4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b2815e6d0 con 0x7f1b2806be90 2024-04-24T23:50:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.391+0000 7f1b2d7f4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f1b280d2ca0 con 0x7f1b2815e2f0 2024-04-24T23:50:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.391+0000 7f1b2d7f4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1b280d2f80 con 0x7f1b280d26e0 2024-04-24T23:50:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.391+0000 7f1b277fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1b2806be90 0x7f1b280cf070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55932/0 (socket says 172.21.15.16:55932) 2024-04-24T23:50:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.391+0000 7f1b277fe640 1 -- 172.21.15.16:0/2914495877 learned_addr learned my addr 172.21.15.16:0/2914495877 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.391+0000 7f1b267fc640 1 --2- 172.21.15.16:0/2914495877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1b280d26e0 0x7f1b2806b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.391+0000 7f1b26ffd640 1 --2- 172.21.15.16:0/2914495877 >> v2:172.21.15.156:3300/0 conn(0x7f1b2815e2f0 0x7f1b2806b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:32.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 -- 172.21.15.16:0/2914495877 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3001369815 0 0) 0x7f1b2815e6d0 con 0x7f1b2806be90 2024-04-24T23:50:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 -- 172.21.15.16:0/2914495877 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b10003520 con 0x7f1b2806be90 2024-04-24T23:50:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 -- 172.21.15.16:0/2914495877 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3434144253 0 0) 0x7f1b10003520 con 0x7f1b2806be90 2024-04-24T23:50:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 -- 172.21.15.16:0/2914495877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1b280d26e0 msgr2=0x7f1b2806b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 --2- 172.21.15.16:0/2914495877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1b280d26e0 0x7f1b2806b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 -- 172.21.15.16:0/2914495877 >> v2:172.21.15.156:3300/0 conn(0x7f1b2815e2f0 msgr2=0x7f1b2806b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 --2- 172.21.15.16:0/2914495877 >> v2:172.21.15.156:3300/0 conn(0x7f1b2815e2f0 0x7f1b2806b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 -- 172.21.15.16:0/2914495877 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b280d3260 con 0x7f1b2806be90 2024-04-24T23:50:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 -- 172.21.15.16:0/2914495877 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f1b180019f0 con 0x7f1b2806be90 2024-04-24T23:50:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 -- 172.21.15.16:0/2914495877 >> v1:172.21.15.16:6789/0 conn(0x7f1b2806be90 legacy=0x7f1b280cf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 --2- 172.21.15.16:0/2914495877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b100039b0 0x7f1b10003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 --2- 172.21.15.16:0/2914495877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b10004800 0x7f1b10004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 --2- 172.21.15.16:0/2914495877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1b280d26e0 0x7f1b100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 -- 172.21.15.16:0/2914495877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1b280d2ca0 con 0x7f1b10004800 2024-04-24T23:50:32.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 -- 172.21.15.16:0/2914495877 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1b280d2f80 con 0x7f1b100039b0 2024-04-24T23:50:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b25ffb640 1 -- 172.21.15.16:0/2914495877 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1b1000b9c0 con 0x7f1b280d26e0 2024-04-24T23:50:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b267fc640 1 --2- 172.21.15.16:0/2914495877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b10004800 0x7f1b10004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b26ffd640 1 --2- 172.21.15.16:0/2914495877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b100039b0 0x7f1b10003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b277fe640 1 --2- 172.21.15.16:0/2914495877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1b280d26e0 0x7f1b100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b267fc640 1 -- 172.21.15.16:0/2914495877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1b280d26e0 msgr2=0x7f1b100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b267fc640 1 --2- 172.21.15.16:0/2914495877 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1b280d26e0 0x7f1b100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b267fc640 1 -- 172.21.15.16:0/2914495877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b100039b0 msgr2=0x7f1b10003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b267fc640 1 --2- 172.21.15.16:0/2914495877 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b100039b0 0x7f1b10003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.392+0000 7f1b267fc640 1 -- 172.21.15.16:0/2914495877 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b10003520 con 0x7f1b10004800 2024-04-24T23:50:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.393+0000 7f1b267fc640 1 --2- 172.21.15.16:0/2914495877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b10004800 0x7f1b10004c80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009ac0 tx=0x7f1b10005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f3137a233b4ab4f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.393+0000 7f1b25ffb640 1 -- 172.21.15.16:0/2914495877 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c036070 con 0x7f1b10004800 2024-04-24T23:50:32.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.393+0000 7f1b25ffb640 1 -- 172.21.15.16:0/2914495877 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b1c031400 con 0x7f1b10004800 2024-04-24T23:50:32.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.393+0000 7f1b25ffb640 1 -- 172.21.15.16:0/2914495877 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b1c009690 con 0x7f1b10004800 2024-04-24T23:50:32.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.394+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/2914495877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b10004800 msgr2=0x7f1b10004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:32.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.394+0000 7f1b2d7f4640 1 --2- 172.21.15.16:0/2914495877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b10004800 0x7f1b10004c80 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c009ac0 tx=0x7f1b10005830 comp rx=0 tx=0).stop 2024-04-24T23:50:32.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.394+0000 7f1b277fe640 1 -- 172.21.15.16:0/2914495877 reap_dead start 2024-04-24T23:50:32.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.394+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/2914495877 shutdown_connections 2024-04-24T23:50:32.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.394+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/2914495877 >> 172.21.15.16:0/2914495877 conn(0x7f1b280d4850 msgr2=0x7f1b280d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:32.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.394+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/2914495877 shutdown_connections 2024-04-24T23:50:32.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.394+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/2914495877 wait complete. 2024-04-24T23:50:32.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.395+0000 7f1b2d7f4640 1 Processor -- start 2024-04-24T23:50:32.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.395+0000 7f1b2d7f4640 1 -- start start 2024-04-24T23:50:32.400 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b2d7f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b10004800 0x7f1b280aa8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:32.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b2d7f4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b28157e80 0x7f1b2815f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:32.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b2d7f4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1b280ccf00 0x7f1b281572b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:32.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b2d7f4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1b280b85d0 con 0x7f1b10004800 2024-04-24T23:50:32.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b2d7f4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1b280b70a0 con 0x7f1b28157e80 2024-04-24T23:50:32.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b2d7f4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1b280b8750 con 0x7f1b280ccf00 2024-04-24T23:50:32.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b26ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b10004800 0x7f1b280aa8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:32.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b26ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b10004800 0x7f1b280aa8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38214/0 (socket says 172.21.15.16:38214) 2024-04-24T23:50:32.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b26ffd640 1 -- 172.21.15.16:0/1905770795 learned_addr learned my addr 172.21.15.16:0/1905770795 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:32.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b267fc640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b28157e80 0x7f1b2815f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:32.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b277fe640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1b280ccf00 0x7f1b281572b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:32.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b26ffd640 1 -- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1b280ccf00 msgr2=0x7f1b281572b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:32.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b26ffd640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1b280ccf00 0x7f1b281572b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:32.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b26ffd640 1 -- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b28157e80 msgr2=0x7f1b2815f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:32.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b26ffd640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b28157e80 0x7f1b2815f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:32.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b26ffd640 1 -- 172.21.15.16:0/1905770795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b28158310 con 0x7f1b10004800 2024-04-24T23:50:32.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b267fc640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b28157e80 0x7f1b2815f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:32.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b277fe640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1b280ccf00 0x7f1b281572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:32.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.396+0000 7f1b26ffd640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b10004800 0x7f1b280aa8e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1b20011960 tx=0x7f1b20011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:32.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.397+0000 7f1b0f7fe640 1 -- 172.21.15.16:0/1905770795 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2000faf0 con 0x7f1b10004800 2024-04-24T23:50:32.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.397+0000 7f1b0f7fe640 1 -- 172.21.15.16:0/1905770795 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b2000fc90 con 0x7f1b10004800 2024-04-24T23:50:32.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.397+0000 7f1b0f7fe640 1 -- 172.21.15.16:0/1905770795 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b20014440 con 0x7f1b10004800 2024-04-24T23:50:32.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.397+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/1905770795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b281585f0 con 0x7f1b10004800 2024-04-24T23:50:32.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.397+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/1905770795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b28158bb0 con 0x7f1b10004800 2024-04-24T23:50:32.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.398+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/1905770795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b28068f90 con 0x7f1b10004800 2024-04-24T23:50:32.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.398+0000 7f1b0f7fe640 1 -- 172.21.15.16:0/1905770795 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1b20012040 con 0x7f1b10004800 2024-04-24T23:50:32.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.399+0000 7f1b0f7fe640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1b1406be30 0x7f1b1406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:32.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.399+0000 7f1b0f7fe640 1 -- 172.21.15.16:0/1905770795 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6187+0+0 (secure 0 0 0) 0x7f1b2008b700 con 0x7f1b10004800 2024-04-24T23:50:32.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.399+0000 7f1b267fc640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1b1406be30 0x7f1b1406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:32.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.399+0000 7f1b267fc640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1b1406be30 0x7f1b1406e2f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c033000 tx=0x7f1b1c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:32.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.403+0000 7f1b0f7fe640 1 -- 172.21.15.16:0/1905770795 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1b20019030 con 0x7f1b10004800 2024-04-24T23:50:32.627 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:32.627+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/1905770795 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) -- 0x7f1b280b63f0 con 0x7f1b10004800 2024-04-24T23:50:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.269+0000 7f1b0f7fe640 1 -- 172.21.15.16:0/1905770795 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v19) ==== 162+0+0 (secure 0 0 0) 0x7f1b20059ed0 con 0x7f1b10004800 2024-04-24T23:50:33.269 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked joinable; MDS may join as newly active. 2024-04-24T23:50:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.273+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1b1406be30 msgr2=0x7f1b1406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.273+0000 7f1b2d7f4640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1b1406be30 0x7f1b1406e2f0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c033000 tx=0x7f1b1c008030 comp rx=0 tx=0).stop 2024-04-24T23:50:33.273 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.273+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/1905770795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b10004800 msgr2=0x7f1b280aa8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.273+0000 7f1b2d7f4640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b10004800 0x7f1b280aa8e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1b20011960 tx=0x7f1b20011e20 comp rx=0 tx=0).stop 2024-04-24T23:50:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.273+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/1905770795 shutdown_connections 2024-04-24T23:50:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.273+0000 7f1b2d7f4640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1b1406be30 0x7f1b1406e2f0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.273+0000 7f1b2d7f4640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1b280ccf00 0x7f1b281572b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.273+0000 7f1b2d7f4640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1b28157e80 0x7f1b2815f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.273+0000 7f1b2d7f4640 1 --2- 172.21.15.16:0/1905770795 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b10004800 0x7f1b280aa8e0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.273+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/1905770795 >> 172.21.15.16:0/1905770795 conn(0x7f1b280d4850 msgr2=0x7f1b280b92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.274+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/1905770795 shutdown_connections 2024-04-24T23:50:33.274 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.274+0000 7f1b2d7f4640 1 -- 172.21.15.16:0/1905770795 wait complete. 2024-04-24T23:50:33.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:50:33.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.397+0000 7f668f8bd640 1 Processor -- start 2024-04-24T23:50:33.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.398+0000 7f668f8bd640 1 -- start start 2024-04-24T23:50:33.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.398+0000 7f668f8bd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f668815e020 0x7f66880cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:33.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.398+0000 7f668f8bd640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f66880b2bb0 0x7f66880cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:33.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.398+0000 7f668f8bd640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66880b3390 con 0x7f66880cfcf0 2024-04-24T23:50:33.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.398+0000 7f668f8bd640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f668806bfc0 con 0x7f66880b2bb0 2024-04-24T23:50:33.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.398+0000 7f668f8bd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f668806c2a0 con 0x7f668815e020 2024-04-24T23:50:33.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.398+0000 7f668de5a640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f66880cfcf0 0x7f6688161bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55944/0 (socket says 172.21.15.16:55944) 2024-04-24T23:50:33.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.398+0000 7f668de5a640 1 -- 172.21.15.16:0/2695593577 learned_addr learned my addr 172.21.15.16:0/2695593577 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:33.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f668d659640 1 --2- 172.21.15.16:0/2695593577 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f668815e020 0x7f66880cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:33.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f668ce58640 1 --2- 172.21.15.16:0/2695593577 >> v2:172.21.15.156:3300/0 conn(0x7f66880b2bb0 0x7f66880cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:33.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 -- 172.21.15.16:0/2695593577 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1455681247 0 0) 0x7f66880b3390 con 0x7f66880cfcf0 2024-04-24T23:50:33.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 -- 172.21.15.16:0/2695593577 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6670003520 con 0x7f66880cfcf0 2024-04-24T23:50:33.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 -- 172.21.15.16:0/2695593577 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 54032215 0 0) 0x7f6670003520 con 0x7f66880cfcf0 2024-04-24T23:50:33.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 -- 172.21.15.16:0/2695593577 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f668815e020 msgr2=0x7f66880cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:33.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 --2- 172.21.15.16:0/2695593577 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f668815e020 0x7f66880cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 -- 172.21.15.16:0/2695593577 >> v2:172.21.15.156:3300/0 conn(0x7f66880b2bb0 msgr2=0x7f66880cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:33.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 --2- 172.21.15.16:0/2695593577 >> v2:172.21.15.156:3300/0 conn(0x7f66880b2bb0 0x7f66880cf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 -- 172.21.15.16:0/2695593577 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668806c580 con 0x7f66880cfcf0 2024-04-24T23:50:33.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 -- 172.21.15.16:0/2695593577 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f66780019f0 con 0x7f66880cfcf0 2024-04-24T23:50:33.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 -- 172.21.15.16:0/2695593577 >> v1:172.21.15.16:6789/0 conn(0x7f66880cfcf0 legacy=0x7f6688161bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:33.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 --2- 172.21.15.16:0/2695593577 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f668815e020 0x7f66700039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:33.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 --2- 172.21.15.16:0/2695593577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670003f30 0x7f6670004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:33.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 --2- 172.21.15.16:0/2695593577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6670004d40 0x7f66700051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:33.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 -- 172.21.15.16:0/2695593577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f668806bfc0 con 0x7f6670003f30 2024-04-24T23:50:33.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 -- 172.21.15.16:0/2695593577 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f668806c2a0 con 0x7f6670004d40 2024-04-24T23:50:33.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f6687fff640 1 -- 172.21.15.16:0/2695593577 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f667000ba10 con 0x7f668815e020 2024-04-24T23:50:33.402 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f668ce58640 1 --2- 172.21.15.16:0/2695593577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670003f30 0x7f6670004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:33.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f668de5a640 1 --2- 172.21.15.16:0/2695593577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6670004d40 0x7f66700051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:33.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.399+0000 7f668d659640 1 --2- 172.21.15.16:0/2695593577 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f668815e020 0x7f66700039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:33.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.400+0000 7f668ce58640 1 -- 172.21.15.16:0/2695593577 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f668815e020 msgr2=0x7f66700039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:33.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.400+0000 7f668ce58640 1 --2- 172.21.15.16:0/2695593577 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f668815e020 0x7f66700039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.400+0000 7f668ce58640 1 -- 172.21.15.16:0/2695593577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6670004d40 msgr2=0x7f66700051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:33.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.400+0000 7f668ce58640 1 --2- 172.21.15.16:0/2695593577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6670004d40 0x7f66700051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.400+0000 7f668ce58640 1 -- 172.21.15.16:0/2695593577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6670003520 con 0x7f6670003f30 2024-04-24T23:50:33.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.400+0000 7f668ce58640 1 --2- 172.21.15.16:0/2695593577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670003f30 0x7f6670004800 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f667c009af0 tx=0x7f6670005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36449428c7a489d1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:33.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.400+0000 7f6687fff640 1 -- 172.21.15.16:0/2695593577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c036070 con 0x7f6670003f30 2024-04-24T23:50:33.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.400+0000 7f6687fff640 1 -- 172.21.15.16:0/2695593577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f667c031400 con 0x7f6670003f30 2024-04-24T23:50:33.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.400+0000 7f6687fff640 1 -- 172.21.15.16:0/2695593577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c009690 con 0x7f6670003f30 2024-04-24T23:50:33.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.401+0000 7f668f8bd640 1 -- 172.21.15.16:0/2695593577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670003f30 msgr2=0x7f6670004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:33.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.401+0000 7f668f8bd640 1 --2- 172.21.15.16:0/2695593577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670003f30 0x7f6670004800 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f667c009af0 tx=0x7f6670005830 comp rx=0 tx=0).stop 2024-04-24T23:50:33.403 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.401+0000 7f668de5a640 1 -- 172.21.15.16:0/2695593577 reap_dead start 2024-04-24T23:50:33.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.401+0000 7f668f8bd640 1 -- 172.21.15.16:0/2695593577 shutdown_connections 2024-04-24T23:50:33.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.401+0000 7f668f8bd640 1 -- 172.21.15.16:0/2695593577 >> 172.21.15.16:0/2695593577 conn(0x7f66880d3e10 msgr2=0x7f66880d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:33.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.401+0000 7f668f8bd640 1 -- 172.21.15.16:0/2695593577 shutdown_connections 2024-04-24T23:50:33.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.401+0000 7f668f8bd640 1 -- 172.21.15.16:0/2695593577 wait complete. 2024-04-24T23:50:33.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.402+0000 7f668f8bd640 1 Processor -- start 2024-04-24T23:50:33.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.402+0000 7f668f8bd640 1 -- start start 2024-04-24T23:50:33.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.403+0000 7f668f8bd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670003f30 0x7f66880c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:33.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.403+0000 7f668f8bd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f668814ad80 0x7f6688149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:33.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.403+0000 7f668f8bd640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6688149760 0x7f668814b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:33.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.403+0000 7f668f8bd640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f668815e400 con 0x7f6670003f30 2024-04-24T23:50:33.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.403+0000 7f668f8bd640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f66880d1a80 con 0x7f6688149760 2024-04-24T23:50:33.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.403+0000 7f668f8bd640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f66880d2fb0 con 0x7f668814ad80 2024-04-24T23:50:33.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.403+0000 7f668d659640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670003f30 0x7f66880c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:33.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.403+0000 7f668d659640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670003f30 0x7f66880c7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38228/0 (socket says 172.21.15.16:38228) 2024-04-24T23:50:33.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.403+0000 7f668d659640 1 -- 172.21.15.16:0/4127178550 learned_addr learned my addr 172.21.15.16:0/4127178550 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:33.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.403+0000 7f668de5a640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6688149760 0x7f668814b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:33.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.404+0000 7f668d659640 1 -- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f668814ad80 msgr2=0x7f6688149220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:33.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.404+0000 7f668ce58640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f668814ad80 0x7f6688149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:33.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.404+0000 7f668d659640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f668814ad80 0x7f6688149220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.404+0000 7f668d659640 1 -- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6688149760 msgr2=0x7f668814b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:33.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.404+0000 7f668d659640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6688149760 0x7f668814b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.404+0000 7f668d659640 1 -- 172.21.15.16:0/4127178550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6688149ce0 con 0x7f6670003f30 2024-04-24T23:50:33.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.404+0000 7f668ce58640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f668814ad80 0x7f6688149220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:33.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.404+0000 7f668de5a640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6688149760 0x7f668814b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:33.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.404+0000 7f668d659640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670003f30 0x7f66880c7020 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f6680011a00 tx=0x7f6680011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:33.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.404+0000 7f6685ffb640 1 -- 172.21.15.16:0/4127178550 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668000fb50 con 0x7f6670003f30 2024-04-24T23:50:33.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.404+0000 7f6685ffb640 1 -- 172.21.15.16:0/4127178550 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f668000fcf0 con 0x7f6670003f30 2024-04-24T23:50:33.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.404+0000 7f6685ffb640 1 -- 172.21.15.16:0/4127178550 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f668000fea0 con 0x7f6670003f30 2024-04-24T23:50:33.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.404+0000 7f668f8bd640 1 -- 172.21.15.16:0/4127178550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6688149fc0 con 0x7f6670003f30 2024-04-24T23:50:33.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.404+0000 7f668f8bd640 1 -- 172.21.15.16:0/4127178550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f668814a580 con 0x7f6670003f30 2024-04-24T23:50:33.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.405+0000 7f668f8bd640 1 -- 172.21.15.16:0/4127178550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f668806dfe0 con 0x7f6670003f30 2024-04-24T23:50:33.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.406+0000 7f6685ffb640 1 -- 172.21.15.16:0/4127178550 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6680012040 con 0x7f6670003f30 2024-04-24T23:50:33.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.406+0000 7f6685ffb640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f667406be30 0x7f667406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:33.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.406+0000 7f6685ffb640 1 -- 172.21.15.16:0/4127178550 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6187+0+0 (secure 0 0 0) 0x7f668008b640 con 0x7f6670003f30 2024-04-24T23:50:33.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.406+0000 7f668ce58640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f667406be30 0x7f667406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:33.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.407+0000 7f668ce58640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f667406be30 0x7f667406e2f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f667c033000 tx=0x7f667c008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:33.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.410+0000 7f6685ffb640 1 -- 172.21.15.16:0/4127178550 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6680019030 con 0x7f6670003f30 2024-04-24T23:50:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.632+0000 7f668f8bd640 1 -- 172.21.15.16:0/4127178550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f668804f5e0 con 0x7f6670003f30 2024-04-24T23:50:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.633+0000 7f6685ffb640 1 -- 172.21.15.16:0/4127178550 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) ==== 94+0+5177 (secure 0 0 0) 0x7f6680059ea0 con 0x7f6670003f30 2024-04-24T23:50:33.632 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:33.633 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":20,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5299,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/448535992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":448535992},{"type":"v1","addr":"172.21.15.16:6835","nonce":448535992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5305,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1890687780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1890687780},{"type":"v1","addr":"172.21.15.156:6836","nonce":1890687780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5308,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2538147395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":2538147395},{"type":"v1","addr":"172.21.15.156:6837","nonce":2538147395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:50:31.671778+0000","modified":"2024-04-24T23:50:33.273095+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5736},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5736":{"gid":5736,"name":"a","rank":0,"incarnation":20,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6834/3643102469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3643102469},{"type":"v1","addr":"172.21.15.16:6834","nonce":3643102469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-24T23:50:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 20 2024-04-24T23:50:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.635+0000 7f668f8bd640 1 -- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f667406be30 msgr2=0x7f667406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.635+0000 7f668f8bd640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f667406be30 0x7f667406e2f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f667c033000 tx=0x7f667c008030 comp rx=0 tx=0).stop 2024-04-24T23:50:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.635+0000 7f668f8bd640 1 -- 172.21.15.16:0/4127178550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670003f30 msgr2=0x7f66880c7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.635+0000 7f668f8bd640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670003f30 0x7f66880c7020 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f6680011a00 tx=0x7f6680011ec0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.635+0000 7f668f8bd640 1 -- 172.21.15.16:0/4127178550 shutdown_connections 2024-04-24T23:50:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.635+0000 7f668f8bd640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f667406be30 0x7f667406e2f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.635+0000 7f668f8bd640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6688149760 0x7f668814b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.635+0000 7f668f8bd640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f668814ad80 0x7f6688149220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.635+0000 7f668f8bd640 1 --2- 172.21.15.16:0/4127178550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6670003f30 0x7f66880c7020 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.635+0000 7f668f8bd640 1 -- 172.21.15.16:0/4127178550 >> 172.21.15.16:0/4127178550 conn(0x7f66880d3e10 msgr2=0x7f66880d1ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.635+0000 7f668f8bd640 1 -- 172.21.15.16:0/4127178550 shutdown_connections 2024-04-24T23:50:33.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:33.635+0000 7f668f8bd640 1 -- 172.21.15.16:0/4127178550 wait complete. 2024-04-24T23:50:34.048 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T23:50:31.671778+0000', 'modified': '2024-04-24T23:50:33.273095+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5736}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5736': {'gid': 5736, 'name': 'a', 'rank': 0, 'incarnation': 20, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.16:6834/3643102469', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6832', 'nonce': 3643102469}, {'type': 'v1', 'addr': '172.21.15.16:6834', 'nonce': 3643102469}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-24T23:50:34.049 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5736:up:replay 2024-04-24T23:50:35.049 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:50:35.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.166+0000 7f8f53112640 1 Processor -- start 2024-04-24T23:50:35.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.166+0000 7f8f53112640 1 -- start start 2024-04-24T23:50:35.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f53112640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f4c144b20 0x7f8f4c144f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:35.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f53112640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f4c145440 con 0x7f8f4c1457b0 2024-04-24T23:50:35.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f53112640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f4c15fe90 con 0x7f8f4c0ced90 2024-04-24T23:50:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f53112640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8f4c160170 con 0x7f8f4c144b20 2024-04-24T23:50:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f4bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8f4c1457b0 0x7f8f4c0ce680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55952/0 (socket says 172.21.15.16:55952) 2024-04-24T23:50:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f4bfff640 1 -- 172.21.15.16:0/668634570 learned_addr learned my addr 172.21.15.16:0/668634570 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f50eae640 1 --2- 172.21.15.16:0/668634570 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f4c144b20 0x7f8f4c144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f4b7fe640 1 -- 172.21.15.16:0/668634570 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 447637869 0 0) 0x7f8f4c145440 con 0x7f8f4c1457b0 2024-04-24T23:50:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f4b7fe640 1 -- 172.21.15.16:0/668634570 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f3c003520 con 0x7f8f4c1457b0 2024-04-24T23:50:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f4b7fe640 1 -- 172.21.15.16:0/668634570 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 445462488 0 0) 0x7f8f3c003520 con 0x7f8f4c1457b0 2024-04-24T23:50:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f4b7fe640 1 -- 172.21.15.16:0/668634570 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f4c144b20 msgr2=0x7f8f4c144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f4b7fe640 1 --2- 172.21.15.16:0/668634570 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f4c144b20 0x7f8f4c144f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f4b7fe640 1 -- 172.21.15.16:0/668634570 >> v1:172.21.15.156:6789/0 conn(0x7f8f4c0ced90 legacy=0x7f8f4c06da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f4b7fe640 1 -- 172.21.15.16:0/668634570 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f4c160450 con 0x7f8f4c1457b0 2024-04-24T23:50:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f4b7fe640 1 -- 172.21.15.16:0/668634570 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f8f38001690 con 0x7f8f4c1457b0 2024-04-24T23:50:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f4b7fe640 1 -- 172.21.15.16:0/668634570 >> v1:172.21.15.16:6789/0 conn(0x7f8f4c1457b0 legacy=0x7f8f4c0ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:35.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.167+0000 7f8f4b7fe640 1 --2- 172.21.15.16:0/668634570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f3c0039e0 0x7f8f3c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.168+0000 7f8f4b7fe640 1 --2- 172.21.15.16:0/668634570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f3c004800 0x7f8f3c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.168+0000 7f8f4b7fe640 1 --2- 172.21.15.16:0/668634570 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f4c144b20 0x7f8f3c005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.168+0000 7f8f4b7fe640 1 -- 172.21.15.16:0/668634570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4c160170 con 0x7f8f3c0039e0 2024-04-24T23:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.168+0000 7f8f4b7fe640 1 -- 172.21.15.16:0/668634570 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8f3c00baf0 con 0x7f8f3c004800 2024-04-24T23:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.168+0000 7f8f4b7fe640 1 -- 172.21.15.16:0/668634570 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8f3c00bcf0 con 0x7f8f4c144b20 2024-04-24T23:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.168+0000 7f8f50eae640 1 --2- 172.21.15.16:0/668634570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f3c0039e0 0x7f8f3c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.168+0000 7f8f4bfff640 1 --2- 172.21.15.16:0/668634570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f3c004800 0x7f8f3c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.168+0000 7f8f516af640 1 --2- 172.21.15.16:0/668634570 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f4c144b20 0x7f8f3c005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.168+0000 7f8f50eae640 1 -- 172.21.15.16:0/668634570 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f4c144b20 msgr2=0x7f8f3c005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.168+0000 7f8f50eae640 1 --2- 172.21.15.16:0/668634570 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f4c144b20 0x7f8f3c005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.168+0000 7f8f50eae640 1 -- 172.21.15.16:0/668634570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f3c004800 msgr2=0x7f8f3c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.168+0000 7f8f50eae640 1 --2- 172.21.15.16:0/668634570 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f3c004800 0x7f8f3c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.168+0000 7f8f50eae640 1 -- 172.21.15.16:0/668634570 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f3c003520 con 0x7f8f3c0039e0 2024-04-24T23:50:35.170 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.168+0000 7f8f516af640 1 --2- 172.21.15.16:0/668634570 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f4c144b20 0x7f8f3c005200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:35.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.169+0000 7f8f50eae640 1 --2- 172.21.15.16:0/668634570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f3c0039e0 0x7f8f3c003e00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009ac0 tx=0x7f8f3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6929362c7ca4227a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:35.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.169+0000 7f8f4b7fe640 1 -- 172.21.15.16:0/668634570 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f40036070 con 0x7f8f3c0039e0 2024-04-24T23:50:35.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.169+0000 7f8f4b7fe640 1 -- 172.21.15.16:0/668634570 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f40031400 con 0x7f8f3c0039e0 2024-04-24T23:50:35.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.169+0000 7f8f4b7fe640 1 -- 172.21.15.16:0/668634570 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f40009690 con 0x7f8f3c0039e0 2024-04-24T23:50:35.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.169+0000 7f8f53112640 1 -- 172.21.15.16:0/668634570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f3c0039e0 msgr2=0x7f8f3c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:35.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.169+0000 7f8f53112640 1 --2- 172.21.15.16:0/668634570 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f3c0039e0 0x7f8f3c003e00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f8f40009ac0 tx=0x7f8f3c005830 comp rx=0 tx=0).stop 2024-04-24T23:50:35.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.169+0000 7f8f516af640 1 -- 172.21.15.16:0/668634570 reap_dead start 2024-04-24T23:50:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.169+0000 7f8f53112640 1 -- 172.21.15.16:0/668634570 shutdown_connections 2024-04-24T23:50:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.169+0000 7f8f53112640 1 -- 172.21.15.16:0/668634570 >> 172.21.15.16:0/668634570 conn(0x7f8f4c0b2000 msgr2=0x7f8f4c0cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.169+0000 7f8f53112640 1 -- 172.21.15.16:0/668634570 shutdown_connections 2024-04-24T23:50:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.169+0000 7f8f53112640 1 -- 172.21.15.16:0/668634570 wait complete. 2024-04-24T23:50:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.171+0000 7f8f53112640 1 Processor -- start 2024-04-24T23:50:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.171+0000 7f8f53112640 1 -- start start 2024-04-24T23:50:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.171+0000 7f8f53112640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f3c0039e0 0x7f8f4c143640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f53112640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f4c1e6800 0x7f8f4c1e6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f53112640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f4c1e71a0 0x7f8f4c159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f53112640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4c0b3390 con 0x7f8f4c1e6800 2024-04-24T23:50:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f53112640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8f4c06d4b0 con 0x7f8f4c1e71a0 2024-04-24T23:50:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f53112640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8f4c15f990 con 0x7f8f3c0039e0 2024-04-24T23:50:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f4bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f4c1e6800 0x7f8f4c1e6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:35.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f4bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f4c1e6800 0x7f8f4c1e6c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:38240/0 (socket says 172.21.15.16:38240) 2024-04-24T23:50:35.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f4bfff640 1 -- 172.21.15.16:0/2495280590 learned_addr learned my addr 172.21.15.16:0/2495280590 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:35.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f50eae640 1 --2- 172.21.15.16:0/2495280590 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f3c0039e0 0x7f8f4c143640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:35.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f516af640 1 --2- 172.21.15.16:0/2495280590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f4c1e71a0 0x7f8f4c159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:35.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f4bfff640 1 -- 172.21.15.16:0/2495280590 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f3c0039e0 msgr2=0x7f8f4c143640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:35.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f4bfff640 1 --2- 172.21.15.16:0/2495280590 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f3c0039e0 0x7f8f4c143640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:35.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f4bfff640 1 -- 172.21.15.16:0/2495280590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f4c1e71a0 msgr2=0x7f8f4c159230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:35.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f4bfff640 1 --2- 172.21.15.16:0/2495280590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f4c1e71a0 0x7f8f4c159230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:35.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f4bfff640 1 -- 172.21.15.16:0/2495280590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f4c1e7990 con 0x7f8f4c1e6800 2024-04-24T23:50:35.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f4bfff640 1 --2- 172.21.15.16:0/2495280590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f4c1e6800 0x7f8f4c1e6c60 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800e8a0 tx=0x7f8f3800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:35.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f497fa640 1 -- 172.21.15.16:0/2495280590 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3800c870 con 0x7f8f4c1e6800 2024-04-24T23:50:35.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f497fa640 1 -- 172.21.15.16:0/2495280590 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f38018070 con 0x7f8f4c1e6800 2024-04-24T23:50:35.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f53112640 1 -- 172.21.15.16:0/2495280590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f4c1e7c50 con 0x7f8f4c1e6800 2024-04-24T23:50:35.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.172+0000 7f8f497fa640 1 -- 172.21.15.16:0/2495280590 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f3800cc30 con 0x7f8f4c1e6800 2024-04-24T23:50:35.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.173+0000 7f8f53112640 1 -- 172.21.15.16:0/2495280590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f4c1e8210 con 0x7f8f4c1e6800 2024-04-24T23:50:35.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.173+0000 7f8f53112640 1 -- 172.21.15.16:0/2495280590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f4c0731f0 con 0x7f8f4c1e6800 2024-04-24T23:50:35.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.174+0000 7f8f497fa640 1 -- 172.21.15.16:0/2495280590 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8f3801d460 con 0x7f8f4c1e6800 2024-04-24T23:50:35.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.174+0000 7f8f497fa640 1 --2- 172.21.15.16:0/2495280590 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8f3006be30 0x7f8f3006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:35.176 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.175+0000 7f8f497fa640 1 -- 172.21.15.16:0/2495280590 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 6187+0+0 (secure 0 0 0) 0x7f8f3808ae10 con 0x7f8f4c1e6800 2024-04-24T23:50:35.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.175+0000 7f8f50eae640 1 --2- 172.21.15.16:0/2495280590 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8f3006be30 0x7f8f3006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:35.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.175+0000 7f8f50eae640 1 --2- 172.21.15.16:0/2495280590 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8f3006be30 0x7f8f3006e2f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8f40033000 tx=0x7f8f40008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:35.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.179+0000 7f8f497fa640 1 -- 172.21.15.16:0/2495280590 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8f38058f00 con 0x7f8f4c1e6800 2024-04-24T23:50:35.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.404+0000 7f8f53112640 1 -- 172.21.15.16:0/2495280590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8f4c04f5e0 con 0x7f8f4c1e6800 2024-04-24T23:50:35.404 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.404+0000 7f8f497fa640 1 -- 172.21.15.16:0/2495280590 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) ==== 94+0+5177 (secure 0 0 0) 0x7f8f3805d200 con 0x7f8f4c1e6800 2024-04-24T23:50:35.404 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:35.404 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":22,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5299,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/448535992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":448535992},{"type":"v1","addr":"172.21.15.16:6835","nonce":448535992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5305,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1890687780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1890687780},{"type":"v1","addr":"172.21.15.156:6836","nonce":1890687780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5308,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2538147395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":2538147395},{"type":"v1","addr":"172.21.15.156:6837","nonce":2538147395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:50:31.671778+0000","modified":"2024-04-24T23:50:34.277421+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5736},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5736":{"gid":5736,"name":"a","rank":0,"incarnation":20,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.16:6834/3643102469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3643102469},{"type":"v1","addr":"172.21.15.16:6834","nonce":3643102469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":0,"qdb_cluster":[]},"id":2}]} 2024-04-24T23:50:35.405 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 22 2024-04-24T23:50:35.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.407+0000 7f8f53112640 1 -- 172.21.15.16:0/2495280590 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8f3006be30 msgr2=0x7f8f3006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.407+0000 7f8f53112640 1 --2- 172.21.15.16:0/2495280590 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8f3006be30 0x7f8f3006e2f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8f40033000 tx=0x7f8f40008030 comp rx=0 tx=0).stop 2024-04-24T23:50:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.407+0000 7f8f53112640 1 -- 172.21.15.16:0/2495280590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f4c1e6800 msgr2=0x7f8f4c1e6c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.407+0000 7f8f53112640 1 --2- 172.21.15.16:0/2495280590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f4c1e6800 0x7f8f4c1e6c60 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8f3800e8a0 tx=0x7f8f3800ed60 comp rx=0 tx=0).stop 2024-04-24T23:50:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.407+0000 7f8f53112640 1 -- 172.21.15.16:0/2495280590 shutdown_connections 2024-04-24T23:50:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.407+0000 7f8f53112640 1 --2- 172.21.15.16:0/2495280590 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8f3006be30 0x7f8f3006e2f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.407+0000 7f8f53112640 1 --2- 172.21.15.16:0/2495280590 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f4c1e71a0 0x7f8f4c159230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.407+0000 7f8f53112640 1 --2- 172.21.15.16:0/2495280590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f4c1e6800 0x7f8f4c1e6c60 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.407+0000 7f8f53112640 1 --2- 172.21.15.16:0/2495280590 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f3c0039e0 0x7f8f4c143640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.407+0000 7f8f53112640 1 -- 172.21.15.16:0/2495280590 >> 172.21.15.16:0/2495280590 conn(0x7f8f4c0b2000 msgr2=0x7f8f4c06d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.407+0000 7f8f53112640 1 -- 172.21.15.16:0/2495280590 shutdown_connections 2024-04-24T23:50:35.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:35.407+0000 7f8f53112640 1 -- 172.21.15.16:0/2495280590 wait complete. 2024-04-24T23:50:35.820 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T23:50:31.671778+0000', 'modified': '2024-04-24T23:50:34.277421+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5736}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5736': {'gid': 5736, 'name': 'a', 'rank': 0, 'incarnation': 20, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.16:6834/3643102469', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6832', 'nonce': 3643102469}, {'type': 'v1', 'addr': '172.21.15.16:6834', 'nonce': 3643102469}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 0, 'qdb_cluster': []} 2024-04-24T23:50:35.820 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_5736:up:rejoin 2024-04-24T23:50:36.270 INFO:tasks.ceph.mds.a.smithi016.stderr:2024-04-24T23:50:36.270+0000 7f82a5994640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T23:50:36.821 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:50:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.934+0000 7f732d5b0640 1 Processor -- start 2024-04-24T23:50:36.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.934+0000 7f732d5b0640 1 -- start start 2024-04-24T23:50:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.935+0000 7f732d5b0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7328144b20 0x7f7328144f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.935+0000 7f732d5b0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7328145440 con 0x7f73281457b0 2024-04-24T23:50:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.935+0000 7f732d5b0640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f732815fe90 con 0x7f73280ced90 2024-04-24T23:50:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.935+0000 7f732d5b0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f7328160170 con 0x7f7328144b20 2024-04-24T23:50:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.935+0000 7f7326ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7328144b20 0x7f7328144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:36.937 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.935+0000 7f73267fc640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f73281457b0 0x7f73280ce680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32848/0 (socket says 172.21.15.16:32848) 2024-04-24T23:50:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.935+0000 7f7326ffd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7328144b20 0x7f7328144f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50726/0 (socket says 172.21.15.16:50726) 2024-04-24T23:50:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.935+0000 7f73267fc640 1 -- 172.21.15.16:0/3695753328 learned_addr learned my addr 172.21.15.16:0/3695753328 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.936+0000 7f7326ffd640 1 -- 172.21.15.16:0/3695753328 >> v1:172.21.15.156:6789/0 conn(0x7f73280ced90 legacy=0x7f732806da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.936+0000 7f7326ffd640 1 -- 172.21.15.16:0/3695753328 >> v1:172.21.15.16:6789/0 conn(0x7f73281457b0 legacy=0x7f73280ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.936+0000 7f7326ffd640 1 -- 172.21.15.16:0/3695753328 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7328160450 con 0x7f7328144b20 2024-04-24T23:50:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.936+0000 7f7325ffb640 1 -- 172.21.15.16:0/3695753328 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2144859884 0 0) 0x7f7328145440 con 0x7f73281457b0 2024-04-24T23:50:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.936+0000 7f7326ffd640 1 --2- 172.21.15.16:0/3695753328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7328144b20 0x7f7328144f00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f73180097a0 tx=0x7f731802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2826006c0d340fb3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.936+0000 7f7325ffb640 1 -- 172.21.15.16:0/3695753328 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7318037070 con 0x7f7328144b20 2024-04-24T23:50:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.936+0000 7f7325ffb640 1 -- 172.21.15.16:0/3695753328 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f731802ae50 con 0x7f7328144b20 2024-04-24T23:50:36.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.936+0000 7f7325ffb640 1 -- 172.21.15.16:0/3695753328 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7318032660 con 0x7f7328144b20 2024-04-24T23:50:36.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.937+0000 7f732d5b0640 1 -- 172.21.15.16:0/3695753328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7328144b20 msgr2=0x7f7328144f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:36.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.937+0000 7f732d5b0640 1 --2- 172.21.15.16:0/3695753328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7328144b20 0x7f7328144f00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f73180097a0 tx=0x7f731802a100 comp rx=0 tx=0).stop 2024-04-24T23:50:36.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.937+0000 7f732d5b0640 1 -- 172.21.15.16:0/3695753328 shutdown_connections 2024-04-24T23:50:36.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.937+0000 7f732d5b0640 1 --2- 172.21.15.16:0/3695753328 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7328144b20 0x7f7328144f00 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:36.942 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.937+0000 7f732d5b0640 1 -- 172.21.15.16:0/3695753328 >> 172.21.15.16:0/3695753328 conn(0x7f73280b2000 msgr2=0x7f73280cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:36.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.937+0000 7f732d5b0640 1 -- 172.21.15.16:0/3695753328 shutdown_connections 2024-04-24T23:50:36.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.937+0000 7f732d5b0640 1 -- 172.21.15.16:0/3695753328 wait complete. 2024-04-24T23:50:36.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.938+0000 7f732d5b0640 1 Processor -- start 2024-04-24T23:50:36.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.938+0000 7f732d5b0640 1 -- start start 2024-04-24T23:50:36.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.939+0000 7f732d5b0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73280ced90 0x7f732813e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:36.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.939+0000 7f732d5b0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7328144b20 0x7f73280cc430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:36.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.939+0000 7f732d5b0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73281457b0 0x7f73280b0210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:36.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.939+0000 7f732d5b0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f73280b3390 con 0x7f73281457b0 2024-04-24T23:50:36.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.939+0000 7f732d5b0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f732806d4b0 con 0x7f73280ced90 2024-04-24T23:50:36.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.939+0000 7f732d5b0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f732815f990 con 0x7f7328144b20 2024-04-24T23:50:36.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.939+0000 7f73277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73281457b0 0x7f73280b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:36.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.939+0000 7f73277fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73281457b0 0x7f73280b0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50738/0 (socket says 172.21.15.16:50738) 2024-04-24T23:50:36.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.939+0000 7f73277fe640 1 -- 172.21.15.16:0/3485740446 learned_addr learned my addr 172.21.15.16:0/3485740446 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:36.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.939+0000 7f7326ffd640 1 --2- 172.21.15.16:0/3485740446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73280ced90 0x7f732813e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:36.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.939+0000 7f73267fc640 1 --2- 172.21.15.16:0/3485740446 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7328144b20 0x7f73280cc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:36.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.940+0000 7f73277fe640 1 -- 172.21.15.16:0/3485740446 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7328144b20 msgr2=0x7f73280cc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:36.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.940+0000 7f73277fe640 1 --2- 172.21.15.16:0/3485740446 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7328144b20 0x7f73280cc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:36.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.940+0000 7f73277fe640 1 -- 172.21.15.16:0/3485740446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73280ced90 msgr2=0x7f732813e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:36.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.940+0000 7f73277fe640 1 --2- 172.21.15.16:0/3485740446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73280ced90 0x7f732813e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:36.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.940+0000 7f73277fe640 1 -- 172.21.15.16:0/3485740446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f732813ebb0 con 0x7f73281457b0 2024-04-24T23:50:36.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.940+0000 7f73277fe640 1 --2- 172.21.15.16:0/3485740446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73281457b0 0x7f73280b0210 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f732000c7a0 tx=0x7f732000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:36.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.940+0000 7f730f7fe640 1 -- 172.21.15.16:0/3485740446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7320003b70 con 0x7f73281457b0 2024-04-24T23:50:36.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.940+0000 7f730f7fe640 1 -- 172.21.15.16:0/3485740446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7320003d30 con 0x7f73281457b0 2024-04-24T23:50:36.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.940+0000 7f732d5b0640 1 -- 172.21.15.16:0/3485740446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73280cc970 con 0x7f73281457b0 2024-04-24T23:50:36.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.940+0000 7f730f7fe640 1 -- 172.21.15.16:0/3485740446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f732000aa90 con 0x7f73281457b0 2024-04-24T23:50:36.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.940+0000 7f732d5b0640 1 -- 172.21.15.16:0/3485740446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73280ccef0 con 0x7f73281457b0 2024-04-24T23:50:36.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.941+0000 7f732d5b0640 1 -- 172.21.15.16:0/3485740446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73280731f0 con 0x7f73281457b0 2024-04-24T23:50:36.945 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.942+0000 7f730f7fe640 1 -- 172.21.15.16:0/3485740446 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f732000ac30 con 0x7f73281457b0 2024-04-24T23:50:36.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.942+0000 7f730f7fe640 1 --2- 172.21.15.16:0/3485740446 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f731406be30 0x7f731406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:36.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.942+0000 7f730f7fe640 1 -- 172.21.15.16:0/3485740446 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 6216+0+0 (secure 0 0 0) 0x7f732008af10 con 0x7f73281457b0 2024-04-24T23:50:36.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.942+0000 7f7326ffd640 1 --2- 172.21.15.16:0/3485740446 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f731406be30 0x7f731406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:36.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.943+0000 7f7326ffd640 1 --2- 172.21.15.16:0/3485740446 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f731406be30 0x7f731406e2f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7318034000 tx=0x7f731802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:36.947 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:36.947+0000 7f730f7fe640 1 -- 172.21.15.16:0/3485740446 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f732813ebb0 con 0x7f73281457b0 2024-04-24T23:50:37.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.170+0000 7f732d5b0640 1 -- 172.21.15.16:0/3485740446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f73280b4e60 con 0x7f73281457b0 2024-04-24T23:50:37.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.170+0000 7f730f7fe640 1 -- 172.21.15.16:0/3485740446 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5184 (secure 0 0 0) 0x7f7320058fd0 con 0x7f73281457b0 2024-04-24T23:50:37.171 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:37.171 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5299,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/448535992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":448535992},{"type":"v1","addr":"172.21.15.16:6835","nonce":448535992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5305,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1890687780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1890687780},{"type":"v1","addr":"172.21.15.156:6836","nonce":1890687780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5308,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2538147395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":2538147395},{"type":"v1","addr":"172.21.15.156:6837","nonce":2538147395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:50:31.671778+0000","modified":"2024-04-24T23:50:35.722577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5736},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5736":{"gid":5736,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6834/3643102469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3643102469},{"type":"v1","addr":"172.21.15.16:6834","nonce":3643102469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":5736,"qdb_cluster":[5736]},"id":2}]} 2024-04-24T23:50:37.172 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 23 2024-04-24T23:50:37.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.173+0000 7f732d5b0640 1 -- 172.21.15.16:0/3485740446 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f731406be30 msgr2=0x7f731406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.173+0000 7f732d5b0640 1 --2- 172.21.15.16:0/3485740446 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f731406be30 0x7f731406e2f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7318034000 tx=0x7f731802e000 comp rx=0 tx=0).stop 2024-04-24T23:50:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.173+0000 7f732d5b0640 1 -- 172.21.15.16:0/3485740446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73281457b0 msgr2=0x7f73280b0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.173+0000 7f732d5b0640 1 --2- 172.21.15.16:0/3485740446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73281457b0 0x7f73280b0210 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f732000c7a0 tx=0x7f732000cc60 comp rx=0 tx=0).stop 2024-04-24T23:50:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.173+0000 7f732d5b0640 1 -- 172.21.15.16:0/3485740446 shutdown_connections 2024-04-24T23:50:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.173+0000 7f732d5b0640 1 --2- 172.21.15.16:0/3485740446 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f731406be30 0x7f731406e2f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.173+0000 7f732d5b0640 1 --2- 172.21.15.16:0/3485740446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f73281457b0 0x7f73280b0210 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.173+0000 7f732d5b0640 1 --2- 172.21.15.16:0/3485740446 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7328144b20 0x7f73280cc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.173+0000 7f732d5b0640 1 --2- 172.21.15.16:0/3485740446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f73280ced90 0x7f732813e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.173+0000 7f732d5b0640 1 -- 172.21.15.16:0/3485740446 >> 172.21.15.16:0/3485740446 conn(0x7f73280b2000 msgr2=0x7f7328071b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.173+0000 7f732d5b0640 1 -- 172.21.15.16:0/3485740446 shutdown_connections 2024-04-24T23:50:37.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.173+0000 7f732d5b0640 1 -- 172.21.15.16:0/3485740446 wait complete. 2024-04-24T23:50:37.588 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-24T23:50:31.671778+0000', 'modified': '2024-04-24T23:50:35.722577+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5736}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5736': {'gid': 5736, 'name': 'a', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.16:6834/3643102469', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.16:6832', 'nonce': 3643102469}, {'type': 'v1', 'addr': '172.21.15.16:6834', 'nonce': 3643102469}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 5736, 'qdb_cluster': [5736]} 2024-04-24T23:50:37.588 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-24T23:50:37.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-04-24T23:50:37.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.692+0000 7f4e7c666640 1 Processor -- start 2024-04-24T23:50:37.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.692+0000 7f4e7c666640 1 -- start start 2024-04-24T23:50:37.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.693+0000 7f4e7c666640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4e7415a450 0x7f4e740b2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.693+0000 7f4e7c666640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e740b3300 con 0x7f4e7406e910 2024-04-24T23:50:37.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.693+0000 7f4e7c666640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e740b35e0 con 0x7f4e740729a0 2024-04-24T23:50:37.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.693+0000 7f4e7c666640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f4e740b38c0 con 0x7f4e7415a450 2024-04-24T23:50:37.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79c01640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4e7406e910 0x7f4e74072290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32854/0 (socket says 172.21.15.16:32854) 2024-04-24T23:50:37.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79c01640 1 -- 172.21.15.16:0/1262140041 learned_addr learned my addr 172.21.15.16:0/1262140041 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:37.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e7a402640 1 --2- 172.21.15.16:0/1262140041 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4e7415a450 0x7f4e740b2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2868089321 0 0) 0x7f4e740b3300 con 0x7f4e7406e910 2024-04-24T23:50:37.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e5c003520 con 0x7f4e7406e910 2024-04-24T23:50:37.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3148510626 0 0) 0x7f4e740b35e0 con 0x7f4e740729a0 2024-04-24T23:50:37.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e740b3300 con 0x7f4e740729a0 2024-04-24T23:50:37.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 27038150 0 0) 0x7f4e5c003520 con 0x7f4e7406e910 2024-04-24T23:50:37.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4e7415a450 msgr2=0x7f4e740b2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 --2- 172.21.15.16:0/1262140041 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4e7415a450 0x7f4e740b2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 >> v1:172.21.15.156:6789/0 conn(0x7f4e740729a0 legacy=0x7f4e740ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e740cc400 con 0x7f4e7406e910 2024-04-24T23:50:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f4e68001670 con 0x7f4e7406e910 2024-04-24T23:50:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 >> v1:172.21.15.16:6789/0 conn(0x7f4e7406e910 legacy=0x7f4e74072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 --2- 172.21.15.16:0/1262140041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e5c003d40 0x7f4e5c008770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 --2- 172.21.15.16:0/1262140041 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4e7415a450 0x7f4e5c00acc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 --2- 172.21.15.16:0/1262140041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e5c00b200 0x7f4e5c00d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e740b38c0 con 0x7f4e5c00b200 2024-04-24T23:50:37.699 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4e5c00dcf0 con 0x7f4e5c003d40 2024-04-24T23:50:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.694+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f4e5c00def0 con 0x7f4e7415a450 2024-04-24T23:50:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.695+0000 7f4e79c01640 1 --2- 172.21.15.16:0/1262140041 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4e7415a450 0x7f4e5c00acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.695+0000 7f4e7ac03640 1 --2- 172.21.15.16:0/1262140041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e5c00b200 0x7f4e5c00d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.695+0000 7f4e7a402640 1 --2- 172.21.15.16:0/1262140041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e5c003d40 0x7f4e5c008770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.695+0000 7f4e7ac03640 1 -- 172.21.15.16:0/1262140041 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4e7415a450 msgr2=0x7f4e5c00acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.695+0000 7f4e7ac03640 1 --2- 172.21.15.16:0/1262140041 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4e7415a450 0x7f4e5c00acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.695+0000 7f4e7ac03640 1 -- 172.21.15.16:0/1262140041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e5c003d40 msgr2=0x7f4e5c008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.695+0000 7f4e7ac03640 1 --2- 172.21.15.16:0/1262140041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e5c003d40 0x7f4e5c008770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.695+0000 7f4e7ac03640 1 -- 172.21.15.16:0/1262140041 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e5c003520 con 0x7f4e5c00b200 2024-04-24T23:50:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.696+0000 7f4e7ac03640 1 --2- 172.21.15.16:0/1262140041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e5c00b200 0x7f4e5c00d5f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4e64001fa0 tx=0x7f4e5c003810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b916a6721b1cfc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:37.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.696+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e64001bf0 con 0x7f4e5c00b200 2024-04-24T23:50:37.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.696+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e64003b70 con 0x7f4e5c00b200 2024-04-24T23:50:37.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.696+0000 7f4e79400640 1 -- 172.21.15.16:0/1262140041 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e640050d0 con 0x7f4e5c00b200 2024-04-24T23:50:37.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.696+0000 7f4e7c666640 1 -- 172.21.15.16:0/1262140041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e5c00b200 msgr2=0x7f4e5c00d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:37.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.696+0000 7f4e7c666640 1 --2- 172.21.15.16:0/1262140041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e5c00b200 0x7f4e5c00d5f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f4e64001fa0 tx=0x7f4e5c003810 comp rx=0 tx=0).stop 2024-04-24T23:50:37.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.696+0000 7f4e7ac03640 1 -- 172.21.15.16:0/1262140041 reap_dead start 2024-04-24T23:50:37.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.696+0000 7f4e7c666640 1 -- 172.21.15.16:0/1262140041 shutdown_connections 2024-04-24T23:50:37.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.696+0000 7f4e7c666640 1 -- 172.21.15.16:0/1262140041 >> 172.21.15.16:0/1262140041 conn(0x7f4e740b2000 msgr2=0x7f4e740cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:37.819 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.697+0000 7f4e7c666640 1 -- 172.21.15.16:0/1262140041 shutdown_connections 2024-04-24T23:50:37.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.697+0000 7f4e7c666640 1 -- 172.21.15.16:0/1262140041 wait complete. 2024-04-24T23:50:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.698+0000 7f4e7c666640 1 Processor -- start 2024-04-24T23:50:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.698+0000 7f4e7c666640 1 -- start start 2024-04-24T23:50:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.698+0000 7f4e7c666640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e5c00b200 0x7f4e741f7a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.698+0000 7f4e7c666640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e7415c530 0x7f4e7415df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e7c666640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4e74154680 0x7f4e74154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e7c666640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e740b2ba0 con 0x7f4e7415c530 2024-04-24T23:50:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e7c666640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4e740b2a20 con 0x7f4e5c00b200 2024-04-24T23:50:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e7c666640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f4e740b2d20 con 0x7f4e74154680 2024-04-24T23:50:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e79c01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e7415c530 0x7f4e7415df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e79c01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e7415c530 0x7f4e7415df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50748/0 (socket says 172.21.15.16:50748) 2024-04-24T23:50:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e79c01640 1 -- 172.21.15.16:0/3772233142 learned_addr learned my addr 172.21.15.16:0/3772233142 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e7ac03640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4e74154680 0x7f4e74154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e7a402640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e5c00b200 0x7f4e741f7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e79c01640 1 -- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4e74154680 msgr2=0x7f4e74154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e79c01640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4e74154680 0x7f4e74154b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e79c01640 1 -- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e5c00b200 msgr2=0x7f4e741f7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e79c01640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e5c00b200 0x7f4e741f7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e79c01640 1 -- 172.21.15.16:0/3772233142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e741f7f80 con 0x7f4e7415c530 2024-04-24T23:50:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e7a402640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e5c00b200 0x7f4e741f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e79c01640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e7415c530 0x7f4e7415df70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4e68007290 tx=0x7f4e68007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e72ffd640 1 -- 172.21.15.16:0/3772233142 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e68001670 con 0x7f4e7415c530 2024-04-24T23:50:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e72ffd640 1 -- 172.21.15.16:0/3772233142 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e68018070 con 0x7f4e7415c530 2024-04-24T23:50:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e7c666640 1 -- 172.21.15.16:0/3772233142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e7415caa0 con 0x7f4e7415c530 2024-04-24T23:50:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e72ffd640 1 -- 172.21.15.16:0/3772233142 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6800fbe0 con 0x7f4e7415c530 2024-04-24T23:50:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.699+0000 7f4e7c666640 1 -- 172.21.15.16:0/3772233142 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e7415d060 con 0x7f4e7415c530 2024-04-24T23:50:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.701+0000 7f4e72ffd640 1 -- 172.21.15.16:0/3772233142 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4e6801d460 con 0x7f4e7415c530 2024-04-24T23:50:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.702+0000 7f4e72ffd640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4e6006be30 0x7f4e6006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.702+0000 7f4e72ffd640 1 -- 172.21.15.16:0/3772233142 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7f4e6808b010 con 0x7f4e7415c530 2024-04-24T23:50:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.702+0000 7f4e7a402640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4e6006be30 0x7f4e6006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.703+0000 7f4e7a402640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4e6006be30 0x7f4e6006e2f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009bc0 tx=0x7f4e6c008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.703+0000 7f4e7f370740 1 Processor -- start 2024-04-24T23:50:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.703+0000 7f4e7f370740 1 -- start start 2024-04-24T23:50:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e7f370740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55ffb60 0x561ea55b24f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e7f370740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55e2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e7f370740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55e2540 0x561ea55b7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e7f370740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ea551af00 con 0x561ea55e2c80 2024-04-24T23:50:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e7f370740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561ea54c3750 con 0x561ea55ffb60 2024-04-24T23:50:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e7f370740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x561ea5516ec0 con 0x561ea55e2540 2024-04-24T23:50:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e79c01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55e2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e79c01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55e2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50762/0 (socket says 172.21.15.16:50762) 2024-04-24T23:50:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e79c01640 1 -- 172.21.15.16:0/1855722139 learned_addr learned my addr 172.21.15.16:0/1855722139 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e7a402640 1 --2- 172.21.15.16:0/1855722139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55ffb60 0x561ea55b24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e7ac03640 1 --2- 172.21.15.16:0/1855722139 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55e2540 0x561ea55b7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e79c01640 1 -- 172.21.15.16:0/1855722139 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55e2540 msgr2=0x561ea55b7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e79c01640 1 --2- 172.21.15.16:0/1855722139 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55e2540 0x561ea55b7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e79c01640 1 -- 172.21.15.16:0/1855722139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55ffb60 msgr2=0x561ea55b24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e79c01640 1 --2- 172.21.15.16:0/1855722139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55ffb60 0x561ea55b24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e79c01640 1 -- 172.21.15.16:0/1855722139 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ea522f2d0 con 0x561ea55e2c80 2024-04-24T23:50:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e79c01640 1 --2- 172.21.15.16:0/1855722139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55e2000 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f4e680892b0 tx=0x7f4e68058a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a05ec476d9f2f1a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e5bfff640 1 -- 172.21.15.16:0/1855722139 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e68071030 con 0x561ea55e2c80 2024-04-24T23:50:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e5bfff640 1 -- 172.21.15.16:0/1855722139 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e68061ca0 con 0x561ea55e2c80 2024-04-24T23:50:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.704+0000 7f4e5bfff640 1 -- 172.21.15.16:0/1855722139 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6800fd80 con 0x561ea55e2c80 2024-04-24T23:50:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.705+0000 7f4e7f370740 1 -- 172.21.15.16:0/1855722139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 msgr2=0x561ea55e2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.705+0000 7f4e7f370740 1 --2- 172.21.15.16:0/1855722139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55e2000 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f4e680892b0 tx=0x7f4e68058a90 comp rx=0 tx=0).stop 2024-04-24T23:50:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.705+0000 7f4e7f370740 1 -- 172.21.15.16:0/1855722139 shutdown_connections 2024-04-24T23:50:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.705+0000 7f4e7f370740 1 --2- 172.21.15.16:0/1855722139 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55e2540 0x561ea55b7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.705+0000 7f4e7f370740 1 --2- 172.21.15.16:0/1855722139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55e2000 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.705+0000 7f4e7f370740 1 --2- 172.21.15.16:0/1855722139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55ffb60 0x561ea55b24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.705+0000 7f4e7f370740 1 -- 172.21.15.16:0/1855722139 >> 172.21.15.16:0/1855722139 conn(0x561ea55b4760 msgr2=0x561ea55fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:37.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.705+0000 7f4e7f370740 1 -- 172.21.15.16:0/1855722139 shutdown_connections 2024-04-24T23:50:37.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.705+0000 7f4e7f370740 1 -- 172.21.15.16:0/1855722139 wait complete. 2024-04-24T23:50:37.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.706+0000 7f4e7f370740 1 -- start start 2024-04-24T23:50:37.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.706+0000 7f4e7f370740 1 Processor -- start 2024-04-24T23:50:37.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7f370740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55fc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7f370740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55ffb60 0x561ea55e3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7f370740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55dd5f0 0x561ea55dda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7f370740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ea55b6910 con 0x561ea55e2c80 2024-04-24T23:50:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7f370740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561ea54d6cf0 con 0x561ea55dd5f0 2024-04-24T23:50:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7f370740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x561ea55b5d00 con 0x561ea55ffb60 2024-04-24T23:50:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7f370740 10 client.0 _ll_register_callbacks cb 0x561ea54d66e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:50:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7a402640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55fc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7a402640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55fc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50768/0 (socket says 172.21.15.16:50768) 2024-04-24T23:50:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7a402640 1 -- 172.21.15.16:0/3123115504 learned_addr learned my addr 172.21.15.16:0/3123115504 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e79c01640 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55ffb60 0x561ea55e3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7ac03640 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55dd5f0 0x561ea55dda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7a402640 1 -- 172.21.15.16:0/3123115504 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55ffb60 msgr2=0x561ea55e3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:37.842 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7a402640 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55ffb60 0x561ea55e3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7a402640 1 -- 172.21.15.16:0/3123115504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55dd5f0 msgr2=0x561ea55dda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7a402640 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55dd5f0 0x561ea55dda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7a402640 1 -- 172.21.15.16:0/3123115504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ea52f96b0 con 0x561ea55e2c80 2024-04-24T23:50:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e79c01640 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55ffb60 0x561ea55e3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7ac03640 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55dd5f0 0x561ea55dda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e7a402640 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55fc000 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c016b50 tx=0x7f4e6c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82db705a1e2ca71 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.707+0000 7f4e597fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-24T23:50:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e597fa640 1 -- 172.21.15.16:0/3123115504 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c023cd0 con 0x561ea55e2c80 2024-04-24T23:50:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e597fa640 1 -- 172.21.15.16:0/3123115504 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7f4e6c02ed20 con 0x561ea55e2c80 2024-04-24T23:50:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e7f370740 1 -- 172.21.15.16:0/3123115504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e68061ca0 con 0x561ea55e2c80 2024-04-24T23:50:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e597fa640 1 -- 172.21.15.16:0/3123115504 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f4e6c023e70 con 0x561ea55e2c80 2024-04-24T23:50:37.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e7f370740 10 client.5787 fetch_fsmap learned FSMap version 23 2024-04-24T23:50:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e7f370740 1 -- 172.21.15.16:0/3123115504 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x561ea55e4720 con 0x561ea55e2c80 2024-04-24T23:50:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e597fa640 1 -- 172.21.15.16:0/3123115504 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e68061ca0 con 0x561ea55e2c80 2024-04-24T23:50:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e597fa640 1 -- 172.21.15.16:0/3123115504 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 3494+0+0 (secure 0 0 0) 0x7f4e6c02d470 con 0x561ea55e2c80 2024-04-24T23:50:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e7f370740 10 client.5787 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T23:50:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e7f370740 10 client.5787 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T23:50:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e7f370740 20 client.5787 populate_metadata read hostname 'smithi016' 2024-04-24T23:50:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e7f370740 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.16:6832/3643102469,v1:172.21.15.16:6834/3643102469] conn(0x561ea55fd350 0x561ea55fd730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e7f370740 4 client.5787 mds_command: new command op to 5736 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T23:50:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e7f370740 1 -- 172.21.15.16:0/3123115504 --> [v2:172.21.15.16:6832/3643102469,v1:172.21.15.16:6834/3643102469] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ea5528440 con 0x561ea55fd350 2024-04-24T23:50:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e79c01640 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.16:6832/3643102469,v1:172.21.15.16:6834/3643102469] conn(0x561ea55fd350 0x561ea55fd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.708+0000 7f4e597fa640 10 client.5787 ms_handle_connect on v2:172.21.15.16:6832/3643102469 2024-04-24T23:50:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.709+0000 7f4e79c01640 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.16:6832/3643102469,v1:172.21.15.16:6834/3643102469] conn(0x561ea55fd350 0x561ea55fd730 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=28d07c3d7bd80f71 server_cookie=c4e215db1a8cbcbf in_seq=0 out_seq=0 2024-04-24T23:50:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.709+0000 7f4e597fa640 1 -- 172.21.15.16:0/3123115504 <== mds.0 v2:172.21.15.16:6832/3643102469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ea5528440 con 0x561ea55fd350 2024-04-24T23:50:37.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.709+0000 7f4e597fa640 10 client.5787 handle_command_reply: tid=0 2024-04-24T23:50:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.709+0000 7f4e7f370740 1 client.5787 shutdown 2024-04-24T23:50:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.709+0000 7f4e7f370740 1 -- 172.21.15.16:0/3123115504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 msgr2=0x561ea55fc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.709+0000 7f4e7f370740 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55fc000 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c016b50 tx=0x7f4e6c004730 comp rx=0 tx=0).stop 2024-04-24T23:50:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.709+0000 7f4e7f370740 1 -- 172.21.15.16:0/3123115504 shutdown_connections 2024-04-24T23:50:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.709+0000 7f4e7f370740 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.16:6832/3643102469,v1:172.21.15.16:6834/3643102469] conn(0x561ea55fd350 0x561ea55fd730 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.709+0000 7f4e7f370740 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55dd5f0 0x561ea55dda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.709+0000 7f4e7f370740 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55ffb60 0x561ea55e3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.709+0000 7f4e7f370740 1 --2- 172.21.15.16:0/3123115504 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55fc000 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.709+0000 7f4e7f370740 1 -- 172.21.15.16:0/3123115504 >> 172.21.15.16:0/3123115504 conn(0x561ea55e2540 msgr2=0x561ea55b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:37.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.709+0000 7f4e597fa640 0 client.5787 ms_handle_reset on v2:172.21.15.16:6832/3643102469 2024-04-24T23:50:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.710+0000 7f4e7f370740 1 -- 172.21.15.16:0/3123115504 shutdown_connections 2024-04-24T23:50:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.710+0000 7f4e7f370740 1 -- 172.21.15.16:0/3123115504 wait complete. 2024-04-24T23:50:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.710+0000 7f4e7f370740 20 client.5787 trim_cache size 0 max 16384 2024-04-24T23:50:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.728+0000 7f4e7f370740 1 Processor -- start 2024-04-24T23:50:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.728+0000 7f4e7f370740 1 -- start start 2024-04-24T23:50:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e7f370740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55dd5f0 0x561ea560cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e7f370740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55b7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e7f370740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55ffb60 0x561ea55b8460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e7f370740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f4e74200460 con 0x561ea55e2c80 2024-04-24T23:50:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e7f370740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741ff7c0 con 0x561ea55ffb60 2024-04-24T23:50:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e7f370740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f4e741ffe10 con 0x561ea55dd5f0 2024-04-24T23:50:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e79c01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55b7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.846 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e79c01640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55b7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50780/0 (socket says 172.21.15.16:50780) 2024-04-24T23:50:37.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e79c01640 1 -- 172.21.15.16:0/158374942 learned_addr learned my addr 172.21.15.16:0/158374942 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:37.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e79c01640 1 -- 172.21.15.16:0/158374942 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55dd5f0 msgr2=0x561ea560cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:37.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e79c01640 1 --2- 172.21.15.16:0/158374942 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55dd5f0 0x561ea560cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e79c01640 1 -- 172.21.15.16:0/158374942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55ffb60 msgr2=0x561ea55b8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:37.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e79c01640 1 --2- 172.21.15.16:0/158374942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55ffb60 0x561ea55b8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e79c01640 1 -- 172.21.15.16:0/158374942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ea55e34e0 con 0x561ea55e2c80 2024-04-24T23:50:37.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e79c01640 1 --2- 172.21.15.16:0/158374942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55b7f20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f4e68001c80 tx=0x7f4e68014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881e608fe433f0cb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:37.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e5b7fe640 1 -- 172.21.15.16:0/158374942 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e68071040 con 0x561ea55e2c80 2024-04-24T23:50:37.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e5b7fe640 1 -- 172.21.15.16:0/158374942 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e68061ca0 con 0x561ea55e2c80 2024-04-24T23:50:37.847 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.729+0000 7f4e5b7fe640 1 -- 172.21.15.16:0/158374942 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e680556a0 con 0x561ea55e2c80 2024-04-24T23:50:37.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.730+0000 7f4e7f370740 1 -- 172.21.15.16:0/158374942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 msgr2=0x561ea55b7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:37.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.730+0000 7f4e7f370740 1 --2- 172.21.15.16:0/158374942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55b7f20 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f4e68001c80 tx=0x7f4e68014000 comp rx=0 tx=0).stop 2024-04-24T23:50:37.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.730+0000 7f4e7f370740 1 -- 172.21.15.16:0/158374942 shutdown_connections 2024-04-24T23:50:37.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.730+0000 7f4e7f370740 1 --2- 172.21.15.16:0/158374942 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55ffb60 0x561ea55b8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.730+0000 7f4e7f370740 1 --2- 172.21.15.16:0/158374942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55e2c80 0x561ea55b7f20 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.730+0000 7f4e7f370740 1 --2- 172.21.15.16:0/158374942 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55dd5f0 0x561ea560cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.730+0000 7f4e7f370740 1 -- 172.21.15.16:0/158374942 >> 172.21.15.16:0/158374942 conn(0x561ea55e2540 msgr2=0x561ea55ad690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:37.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.730+0000 7f4e7f370740 1 -- 172.21.15.16:0/158374942 shutdown_connections 2024-04-24T23:50:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.730+0000 7f4e7f370740 1 -- 172.21.15.16:0/158374942 wait complete. 2024-04-24T23:50:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.730+0000 7f4e7f370740 1 -- start start 2024-04-24T23:50:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.730+0000 7f4e7f370740 1 Processor -- start 2024-04-24T23:50:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.731+0000 7f4e7f370740 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55dd5f0 0x561ea55f2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.731+0000 7f4e7f370740 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55e2c80 0x561ea55f3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.731+0000 7f4e7f370740 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55ffb60 0x561ea55f8980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.731+0000 7f4e7f370740 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x561ea55f9b80 con 0x561ea55dd5f0 2024-04-24T23:50:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.731+0000 7f4e7f370740 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x561ea560d5c0 con 0x561ea55ffb60 2024-04-24T23:50:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.731+0000 7f4e7f370740 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x561ea55e5110 con 0x561ea55e2c80 2024-04-24T23:50:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.731+0000 7f4e7f370740 10 client.0 _ll_register_callbacks cb 0x7f4e6c016430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T23:50:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.731+0000 7f4e7a402640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55dd5f0 0x561ea55f2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.731+0000 7f4e7a402640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55dd5f0 0x561ea55f2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50794/0 (socket says 172.21.15.16:50794) 2024-04-24T23:50:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.731+0000 7f4e7a402640 1 -- 172.21.15.16:0/1071712864 learned_addr learned my addr 172.21.15.16:0/1071712864 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:37.849 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e7ac03640 1 --2- 172.21.15.16:0/1071712864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55ffb60 0x561ea55f8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e79c01640 1 --2- 172.21.15.16:0/1071712864 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55e2c80 0x561ea55f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e7a402640 1 -- 172.21.15.16:0/1071712864 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55e2c80 msgr2=0x561ea55f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:37.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e7a402640 1 --2- 172.21.15.16:0/1071712864 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55e2c80 0x561ea55f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e7a402640 1 -- 172.21.15.16:0/1071712864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55ffb60 msgr2=0x561ea55f8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:37.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e7a402640 1 --2- 172.21.15.16:0/1071712864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55ffb60 0x561ea55f8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e7a402640 1 -- 172.21.15.16:0/1071712864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ea55f3650 con 0x561ea55dd5f0 2024-04-24T23:50:37.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e7a402640 1 --2- 172.21.15.16:0/1071712864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55dd5f0 0x561ea55f2b50 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009970 tx=0x7f4e6c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5433165a7b3622a0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:37.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e597fa640 10 client.0 ms_handle_connect on v2:172.21.15.16:3300/0 2024-04-24T23:50:37.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e597fa640 1 -- 172.21.15.16:0/1071712864 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c014860 con 0x561ea55dd5f0 2024-04-24T23:50:37.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e597fa640 1 -- 172.21.15.16:0/1071712864 <== mon.0 v2:172.21.15.16:3300/0 2 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7f4e6c042cd0 con 0x561ea55dd5f0 2024-04-24T23:50:37.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e7f370740 1 -- 172.21.15.16:0/1071712864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e68061ca0 con 0x561ea55dd5f0 2024-04-24T23:50:37.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e597fa640 1 -- 172.21.15.16:0/1071712864 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f4e6c004040 con 0x561ea55dd5f0 2024-04-24T23:50:37.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e7f370740 10 client.5793 fetch_fsmap learned FSMap version 23 2024-04-24T23:50:37.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e7f370740 1 -- 172.21.15.16:0/1071712864 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x561ea55de610 con 0x561ea55dd5f0 2024-04-24T23:50:37.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e597fa640 1 -- 172.21.15.16:0/1071712864 <== mon.0 v2:172.21.15.16:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e68061ca0 con 0x561ea55dd5f0 2024-04-24T23:50:37.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.732+0000 7f4e597fa640 1 -- 172.21.15.16:0/1071712864 <== mon.0 v2:172.21.15.16:3300/0 5 ==== fsmap(e 23) ==== 3494+0+0 (secure 0 0 0) 0x7f4e6c049050 con 0x561ea55dd5f0 2024-04-24T23:50:37.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 10 client.5793 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T23:50:37.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 10 client.5793 resolve_mds: resolved name 'a' to daemon mds.a 2024-04-24T23:50:37.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 20 client.5793 populate_metadata read hostname 'smithi016' 2024-04-24T23:50:37.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 1 --2- 172.21.15.16:0/1071712864 >> [v2:172.21.15.16:6832/3643102469,v1:172.21.15.16:6834/3643102469] conn(0x561ea561fc00 0x561ea561ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:37.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 4 client.5793 mds_command: new command op to 5736 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T23:50:37.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 1 -- 172.21.15.16:0/1071712864 --> [v2:172.21.15.16:6832/3643102469,v1:172.21.15.16:6834/3643102469] -- command(tid 0: {"prefix": "status"}) -- 0x561ea55b3bb0 con 0x561ea561fc00 2024-04-24T23:50:37.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e79c01640 1 --2- 172.21.15.16:0/1071712864 >> [v2:172.21.15.16:6832/3643102469,v1:172.21.15.16:6834/3643102469] conn(0x561ea561fc00 0x561ea561ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:37.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e597fa640 10 client.5793 ms_handle_connect on v2:172.21.15.16:6832/3643102469 2024-04-24T23:50:37.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e79c01640 1 --2- 172.21.15.16:0/1071712864 >> [v2:172.21.15.16:6832/3643102469,v1:172.21.15.16:6834/3643102469] conn(0x561ea561fc00 0x561ea561ffe0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=393bfd6fbe085062 server_cookie=3d87da90ce1e2354 in_seq=0 out_seq=0 2024-04-24T23:50:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e597fa640 1 -- 172.21.15.16:0/1071712864 <== mds.0 v2:172.21.15.16:6832/3643102469 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x561ea55b3bb0 con 0x561ea561fc00 2024-04-24T23:50:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e597fa640 10 client.5793 handle_command_reply: tid=0 2024-04-24T23:50:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 1 client.5793 shutdown 2024-04-24T23:50:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 1 -- 172.21.15.16:0/1071712864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55dd5f0 msgr2=0x561ea55f2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 1 --2- 172.21.15.16:0/1071712864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55dd5f0 0x561ea55f2b50 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009970 tx=0x7f4e6c009190 comp rx=0 tx=0).stop 2024-04-24T23:50:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 1 -- 172.21.15.16:0/1071712864 shutdown_connections 2024-04-24T23:50:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 1 --2- 172.21.15.16:0/1071712864 >> [v2:172.21.15.16:6832/3643102469,v1:172.21.15.16:6834/3643102469] conn(0x561ea561fc00 0x561ea561ffe0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 1 --2- 172.21.15.16:0/1071712864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x561ea55ffb60 0x561ea55f8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e597fa640 0 client.5793 ms_handle_reset on v2:172.21.15.16:6832/3643102469 2024-04-24T23:50:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 1 --2- 172.21.15.16:0/1071712864 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x561ea55e2c80 0x561ea55f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 1 --2- 172.21.15.16:0/1071712864 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x561ea55dd5f0 0x561ea55f2b50 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.733+0000 7f4e7f370740 1 -- 172.21.15.16:0/1071712864 >> 172.21.15.16:0/1071712864 conn(0x561ea55e2540 msgr2=0x561ea560fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7f370740 1 -- 172.21.15.16:0/1071712864 shutdown_connections 2024-04-24T23:50:37.852 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7f370740 1 -- 172.21.15.16:0/1071712864 wait complete. 2024-04-24T23:50:37.853 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7f370740 20 client.5793 trim_cache size 0 max 16384 2024-04-24T23:50:37.853 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:50:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "f8999c8c-ab57-4aba-8d53-08e19e6ee3d5", 2024-04-24T23:50:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2024-04-24T23:50:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 5736, 2024-04-24T23:50:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2024-04-24T23:50:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2024-04-24T23:50:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: "fs_name": "cephfs", 2024-04-24T23:50:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 4.4605287599999999, 2024-04-24T23:50:37.853 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 23, 2024-04-24T23:50:37.854 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 43, 2024-04-24T23:50:37.854 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 41, 2024-04-24T23:50:37.854 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 7.4418261770000003 2024-04-24T23:50:37.854 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:50:37.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7c666640 1 -- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4e6006be30 msgr2=0x7f4e6006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:37.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7c666640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4e6006be30 0x7f4e6006e2f0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009bc0 tx=0x7f4e6c008040 comp rx=0 tx=0).stop 2024-04-24T23:50:37.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7c666640 1 -- 172.21.15.16:0/3772233142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e7415c530 msgr2=0x7f4e7415df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:37.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7c666640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e7415c530 0x7f4e7415df70 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f4e68007290 tx=0x7f4e68007750 comp rx=0 tx=0).stop 2024-04-24T23:50:37.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7c666640 1 -- 172.21.15.16:0/3772233142 shutdown_connections 2024-04-24T23:50:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7c666640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f4e6006be30 0x7f4e6006e2f0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7c666640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f4e74154680 0x7f4e74154b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7c666640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4e7415c530 0x7f4e7415df70 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7c666640 1 --2- 172.21.15.16:0/3772233142 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e5c00b200 0x7f4e741f7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7c666640 1 -- 172.21.15.16:0/3772233142 >> 172.21.15.16:0/3772233142 conn(0x7f4e740b2000 msgr2=0x7f4e741526d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7c666640 1 -- 172.21.15.16:0/3772233142 shutdown_connections 2024-04-24T23:50:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:37.734+0000 7f4e7c666640 1 -- 172.21.15.16:0/3772233142 wait complete. 2024-04-24T23:50:37.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:37.856 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest/mnt.0 && exec cat /home/ubuntu/cephtest/mnt.0/file_on_fs_0) 2024-04-24T23:50:37.878 INFO:teuthology.orchestra.run.smithi016.stdout:0 2024-04-24T23:50:37.879 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-04-24T23:50:37.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T23:50:37.962 DEBUG:teuthology.orchestra.run.smithi156:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-04-24T23:50:38.013 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-04-24T23:50:38.013 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T23:50:38.014 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:50:38.014 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:50:38.030 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:50:38.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.171+0000 7f9299330640 1 Processor -- start 2024-04-24T23:50:38.172 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.171+0000 7f9299330640 1 -- start start 2024-04-24T23:50:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f9299330640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9294071a00 0x7f9294071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f9299330640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9294144740 0x7f929415c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f9299330640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9294072320 con 0x7f929415c8e0 2024-04-24T23:50:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f9299330640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f929415c1f0 con 0x7f9294144740 2024-04-24T23:50:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f9299330640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f92940cf3a0 con 0x7f9294071a00 2024-04-24T23:50:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f928bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9294144740 0x7f929415c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f92937fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f929415c8e0 0x7f92940ce7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32862/0 (socket says 172.21.15.16:32862) 2024-04-24T23:50:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f92937fe640 1 -- 172.21.15.16:0/1647489578 learned_addr learned my addr 172.21.15.16:0/1647489578 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f9292ffd640 1 --2- 172.21.15.16:0/1647489578 >> v2:172.21.15.156:3300/0 conn(0x7f9294071a00 0x7f9294071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f92927fc640 1 -- 172.21.15.16:0/1647489578 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1419956853 0 0) 0x7f9294072320 con 0x7f929415c8e0 2024-04-24T23:50:38.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f92927fc640 1 -- 172.21.15.16:0/1647489578 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9280005730 con 0x7f929415c8e0 2024-04-24T23:50:38.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f928bfff640 1 -- 172.21.15.16:0/1647489578 >> v2:172.21.15.156:3300/0 conn(0x7f9294071a00 msgr2=0x7f9294071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:38.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f928bfff640 1 --2- 172.21.15.16:0/1647489578 >> v2:172.21.15.156:3300/0 conn(0x7f9294071a00 0x7f9294071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f928bfff640 1 -- 172.21.15.16:0/1647489578 >> v1:172.21.15.16:6789/0 conn(0x7f929415c8e0 legacy=0x7f92940ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.172+0000 7f928bfff640 1 -- 172.21.15.16:0/1647489578 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92940cc8d0 con 0x7f9294144740 2024-04-24T23:50:38.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.173+0000 7f928bfff640 1 --2- 172.21.15.16:0/1647489578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294144740 0x7f929415c3a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f92940b4d20 tx=0x7f927c02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3883c9ea466c1a8e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:38.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.173+0000 7f92927fc640 1 -- 172.21.15.16:0/1647489578 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927c037070 con 0x7f9294144740 2024-04-24T23:50:38.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.173+0000 7f92927fc640 1 -- 172.21.15.16:0/1647489578 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f927c032440 con 0x7f9294144740 2024-04-24T23:50:38.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.173+0000 7f92927fc640 1 -- 172.21.15.16:0/1647489578 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f927c032810 con 0x7f9294144740 2024-04-24T23:50:38.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.174+0000 7f9299330640 1 -- 172.21.15.16:0/1647489578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294144740 msgr2=0x7f929415c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.174+0000 7f9299330640 1 --2- 172.21.15.16:0/1647489578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294144740 0x7f929415c3a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f92940b4d20 tx=0x7f927c02a1a0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.177 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.174+0000 7f9299330640 1 -- 172.21.15.16:0/1647489578 shutdown_connections 2024-04-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.174+0000 7f9299330640 1 --2- 172.21.15.16:0/1647489578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294144740 0x7f929415c3a0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.174+0000 7f9299330640 1 --2- 172.21.15.16:0/1647489578 >> v2:172.21.15.156:3300/0 conn(0x7f9294071a00 0x7f9294071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.174+0000 7f9299330640 1 -- 172.21.15.16:0/1647489578 >> 172.21.15.16:0/1647489578 conn(0x7f92940b2000 msgr2=0x7f92940cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.174+0000 7f9299330640 1 -- 172.21.15.16:0/1647489578 shutdown_connections 2024-04-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.174+0000 7f9299330640 1 -- 172.21.15.16:0/1647489578 wait complete. 2024-04-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.175+0000 7f9299330640 1 Processor -- start 2024-04-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.175+0000 7f9299330640 1 -- start start 2024-04-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f9299330640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9294071a00 0x7f92941f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f9299330640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f929415c8e0 0x7f929413a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f9299330640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294159d40 0x7f92940b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f9299330640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f92940cd170 con 0x7f9294159d40 2024-04-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f9299330640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f92940cce90 con 0x7f9294071a00 2024-04-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f9299330640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f92940ceed0 con 0x7f929415c8e0 2024-04-24T23:50:38.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f92937fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294159d40 0x7f92940b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f92937fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294159d40 0x7f92940b3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50812/0 (socket says 172.21.15.16:50812) 2024-04-24T23:50:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f92937fe640 1 -- 172.21.15.16:0/4002946882 learned_addr learned my addr 172.21.15.16:0/4002946882 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f92937fe640 1 -- 172.21.15.16:0/4002946882 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f929415c8e0 msgr2=0x7f929413a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f92937fe640 1 --2- 172.21.15.16:0/4002946882 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f929415c8e0 0x7f929413a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f92937fe640 1 -- 172.21.15.16:0/4002946882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9294071a00 msgr2=0x7f92941f3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f92937fe640 1 --2- 172.21.15.16:0/4002946882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9294071a00 0x7f92941f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f92937fe640 1 -- 172.21.15.16:0/4002946882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929413abc0 con 0x7f9294159d40 2024-04-24T23:50:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f92937fe640 1 --2- 172.21.15.16:0/4002946882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294159d40 0x7f92940b3a30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f928c00cad0 tx=0x7f928c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.176+0000 7f928b7fe640 1 -- 172.21.15.16:0/4002946882 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c017070 con 0x7f9294159d40 2024-04-24T23:50:38.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.177+0000 7f928b7fe640 1 -- 172.21.15.16:0/4002946882 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f928c00ac30 con 0x7f9294159d40 2024-04-24T23:50:38.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.177+0000 7f9299330640 1 -- 172.21.15.16:0/4002946882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929413c540 con 0x7f9294159d40 2024-04-24T23:50:38.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.177+0000 7f928b7fe640 1 -- 172.21.15.16:0/4002946882 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928c012400 con 0x7f9294159d40 2024-04-24T23:50:38.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.177+0000 7f9299330640 1 -- 172.21.15.16:0/4002946882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f929413cb00 con 0x7f9294159d40 2024-04-24T23:50:38.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.178+0000 7f9299330640 1 -- 172.21.15.16:0/4002946882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92940731f0 con 0x7f9294159d40 2024-04-24T23:50:38.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.178+0000 7f928b7fe640 1 -- 172.21.15.16:0/4002946882 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f928c01a070 con 0x7f9294159d40 2024-04-24T23:50:38.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.179+0000 7f928b7fe640 1 --2- 172.21.15.16:0/4002946882 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f927406be30 0x7f927406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.179+0000 7f928b7fe640 1 -- 172.21.15.16:0/4002946882 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7f928c08adb0 con 0x7f9294159d40 2024-04-24T23:50:38.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.179+0000 7f9292ffd640 1 --2- 172.21.15.16:0/4002946882 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f927406be30 0x7f927406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.181 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.179+0000 7f9292ffd640 1 --2- 172.21.15.16:0/4002946882 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f927406be30 0x7f927406e2f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f92840098f0 tx=0x7f9284008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:38.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.183+0000 7f928b7fe640 1 -- 172.21.15.16:0/4002946882 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f928c0595f0 con 0x7f9294159d40 2024-04-24T23:50:38.370 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.369+0000 7f9299330640 1 -- 172.21.15.16:0/4002946882 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f92940695b0 con 0x7f9294159d40 2024-04-24T23:50:38.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.370+0000 7f928b7fe640 1 -- 172.21.15.16:0/4002946882 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+17050 (secure 0 0 0) 0x7f928c058ef0 con 0x7f9294159d40 2024-04-24T23:50:38.371 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:38.371 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":43,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:50:37.287284+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:50:17.026573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-24T23:50:17.415963+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.36","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6834/561619268":"2024-04-25T23:50:30.259891+0000","172.21.15.16:6832/561619268":"2024-04-25T23:50:30.259891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:50:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.372+0000 7f9299330640 1 -- 172.21.15.16:0/4002946882 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f927406be30 msgr2=0x7f927406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.372+0000 7f9299330640 1 --2- 172.21.15.16:0/4002946882 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f927406be30 0x7f927406e2f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f92840098f0 tx=0x7f9284008040 comp rx=0 tx=0).stop 2024-04-24T23:50:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.372+0000 7f9299330640 1 -- 172.21.15.16:0/4002946882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294159d40 msgr2=0x7f92940b3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.372+0000 7f9299330640 1 --2- 172.21.15.16:0/4002946882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294159d40 0x7f92940b3a30 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f928c00cad0 tx=0x7f928c00cf90 comp rx=0 tx=0).stop 2024-04-24T23:50:38.373 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.372+0000 7f9299330640 1 -- 172.21.15.16:0/4002946882 shutdown_connections 2024-04-24T23:50:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.372+0000 7f9299330640 1 --2- 172.21.15.16:0/4002946882 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f927406be30 0x7f927406e2f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.372+0000 7f9299330640 1 --2- 172.21.15.16:0/4002946882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294159d40 0x7f92940b3a30 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.372+0000 7f9299330640 1 --2- 172.21.15.16:0/4002946882 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f929415c8e0 0x7f929413a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.372+0000 7f9299330640 1 --2- 172.21.15.16:0/4002946882 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9294071a00 0x7f92941f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.372+0000 7f9299330640 1 -- 172.21.15.16:0/4002946882 >> 172.21.15.16:0/4002946882 conn(0x7f92940b2000 msgr2=0x7f9294071de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.372+0000 7f9299330640 1 -- 172.21.15.16:0/4002946882 shutdown_connections 2024-04-24T23:50:38.374 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.372+0000 7f9299330640 1 -- 172.21.15.16:0/4002946882 wait complete. 2024-04-24T23:50:38.384 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:50:17.026573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:50:17.415963+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T23:50:38.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:50:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.487+0000 7ff245645640 1 Processor -- start 2024-04-24T23:50:38.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.487+0000 7ff245645640 1 -- start start 2024-04-24T23:50:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.488+0000 7ff245645640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff2400af980 0x7ff24006deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.488+0000 7ff245645640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff24006d8d0 con 0x7ff24006d360 2024-04-24T23:50:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.488+0000 7ff245645640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff24006dbb0 con 0x7ff24015dff0 2024-04-24T23:50:38.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.488+0000 7ff245645640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff2400732d0 con 0x7ff2400af980 2024-04-24T23:50:38.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.488+0000 7ff23e7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff2400af980 0x7ff24006deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.488+0000 7ff23f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff24006d360 0x7ff240071e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32872/0 (socket says 172.21.15.16:32872) 2024-04-24T23:50:38.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.489+0000 7ff23f7fe640 1 -- 172.21.15.16:0/3086169977 learned_addr learned my addr 172.21.15.16:0/3086169977 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:38.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.489+0000 7ff23e7fc640 1 -- 172.21.15.16:0/3086169977 >> v1:172.21.15.156:6789/0 conn(0x7ff24015dff0 legacy=0x7ff2400af270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:38.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.489+0000 7ff23e7fc640 1 -- 172.21.15.16:0/3086169977 >> v1:172.21.15.16:6789/0 conn(0x7ff24006d360 legacy=0x7ff240071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.490 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.489+0000 7ff23e7fc640 1 -- 172.21.15.16:0/3086169977 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff240071330 con 0x7ff2400af980 2024-04-24T23:50:38.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.489+0000 7ff23e7fc640 1 --2- 172.21.15.16:0/3086169977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2400af980 0x7ff24006deb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff2280097a0 tx=0x7ff22802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a629a55f59cc4d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:38.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.490+0000 7ff23dffb640 1 -- 172.21.15.16:0/3086169977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff228037070 con 0x7ff2400af980 2024-04-24T23:50:38.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.490+0000 7ff23dffb640 1 -- 172.21.15.16:0/3086169977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff228032440 con 0x7ff2400af980 2024-04-24T23:50:38.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.490+0000 7ff23dffb640 1 -- 172.21.15.16:0/3086169977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff228032810 con 0x7ff2400af980 2024-04-24T23:50:38.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.490+0000 7ff245645640 1 -- 172.21.15.16:0/3086169977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2400af980 msgr2=0x7ff24006deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.491 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.490+0000 7ff245645640 1 --2- 172.21.15.16:0/3086169977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2400af980 0x7ff24006deb0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff2280097a0 tx=0x7ff22802a1e0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.490+0000 7ff245645640 1 -- 172.21.15.16:0/3086169977 shutdown_connections 2024-04-24T23:50:38.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.490+0000 7ff245645640 1 --2- 172.21.15.16:0/3086169977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff2400af980 0x7ff24006deb0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.490+0000 7ff245645640 1 -- 172.21.15.16:0/3086169977 >> 172.21.15.16:0/3086169977 conn(0x7ff2400affc0 msgr2=0x7ff2400b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:38.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.490+0000 7ff245645640 1 -- 172.21.15.16:0/3086169977 shutdown_connections 2024-04-24T23:50:38.492 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.490+0000 7ff245645640 1 -- 172.21.15.16:0/3086169977 wait complete. 2024-04-24T23:50:38.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.492+0000 7ff245645640 1 Processor -- start 2024-04-24T23:50:38.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.492+0000 7ff245645640 1 -- start start 2024-04-24T23:50:38.494 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.492+0000 7ff245645640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff24006d360 0x7ff240152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.492+0000 7ff245645640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2400af980 0x7ff24015e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.492+0000 7ff245645640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff24015dff0 0x7ff2400ce050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.492+0000 7ff245645640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff2400ccca0 con 0x7ff24006d360 2024-04-24T23:50:38.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.492+0000 7ff245645640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff2400718f0 con 0x7ff2400af980 2024-04-24T23:50:38.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.492+0000 7ff245645640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff2400cce20 con 0x7ff24015dff0 2024-04-24T23:50:38.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.492+0000 7ff23effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff24006d360 0x7ff240152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff23effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff24006d360 0x7ff240152ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50836/0 (socket says 172.21.15.16:50836) 2024-04-24T23:50:38.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff23effd640 1 -- 172.21.15.16:0/1664043661 learned_addr learned my addr 172.21.15.16:0/1664043661 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:38.495 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff23f7fe640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff24015dff0 0x7ff2400ce050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff23e7fc640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2400af980 0x7ff24015e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff23effd640 1 -- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff24015dff0 msgr2=0x7ff2400ce050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff23effd640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff24015dff0 0x7ff2400ce050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff23effd640 1 -- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2400af980 msgr2=0x7ff24015e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff23effd640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2400af980 0x7ff24015e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff23effd640 1 -- 172.21.15.16:0/1664043661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24015ee50 con 0x7ff24006d360 2024-04-24T23:50:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff23e7fc640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2400af980 0x7ff24015e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff23effd640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff24006d360 0x7ff240152ed0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff23800b420 tx=0x7ff23800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff2377fe640 1 -- 172.21.15.16:0/1664043661 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff238011830 con 0x7ff24006d360 2024-04-24T23:50:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff2377fe640 1 -- 172.21.15.16:0/1664043661 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff238016070 con 0x7ff24006d360 2024-04-24T23:50:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff245645640 1 -- 172.21.15.16:0/1664043661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff24015f130 con 0x7ff24006d360 2024-04-24T23:50:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff2377fe640 1 -- 172.21.15.16:0/1664043661 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff238011c10 con 0x7ff24006d360 2024-04-24T23:50:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.493+0000 7ff245645640 1 -- 172.21.15.16:0/1664043661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff24015f6f0 con 0x7ff24006d360 2024-04-24T23:50:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.494+0000 7ff245645640 1 -- 172.21.15.16:0/1664043661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff240068f90 con 0x7ff24006d360 2024-04-24T23:50:38.497 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.495+0000 7ff2377fe640 1 -- 172.21.15.16:0/1664043661 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff23801b460 con 0x7ff24006d360 2024-04-24T23:50:38.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.495+0000 7ff2377fe640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff22006be30 0x7ff22006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.495+0000 7ff2377fe640 1 -- 172.21.15.16:0/1664043661 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7ff23808aa90 con 0x7ff24006d360 2024-04-24T23:50:38.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.496+0000 7ff23e7fc640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff22006be30 0x7ff22006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.498 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.496+0000 7ff23e7fc640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff22006be30 0x7ff22006e2f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff228034000 tx=0x7ff22802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:38.500 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.500+0000 7ff2377fe640 1 -- 172.21.15.16:0/1664043661 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff238020020 con 0x7ff24006d360 2024-04-24T23:50:38.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.680+0000 7ff245645640 1 -- 172.21.15.16:0/1664043661 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff240161730 con 0x7ff24006d360 2024-04-24T23:50:38.682 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.681+0000 7ff2377fe640 1 -- 172.21.15.16:0/1664043661 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) ==== 93+0+10 (secure 0 0 0) 0x7ff2380592d0 con 0x7ff24006d360 2024-04-24T23:50:38.682 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:50:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.683+0000 7ff245645640 1 -- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff22006be30 msgr2=0x7ff22006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.683+0000 7ff245645640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff22006be30 0x7ff22006e2f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff228034000 tx=0x7ff22802e000 comp rx=0 tx=0).stop 2024-04-24T23:50:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.683+0000 7ff245645640 1 -- 172.21.15.16:0/1664043661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff24006d360 msgr2=0x7ff240152ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.683+0000 7ff245645640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff24006d360 0x7ff240152ed0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff23800b420 tx=0x7ff23800b8e0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.683+0000 7ff245645640 1 -- 172.21.15.16:0/1664043661 shutdown_connections 2024-04-24T23:50:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.683+0000 7ff245645640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff22006be30 0x7ff22006e2f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.683+0000 7ff245645640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff24015dff0 0x7ff2400ce050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.683+0000 7ff245645640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff2400af980 0x7ff24015e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.683+0000 7ff245645640 1 --2- 172.21.15.16:0/1664043661 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff24006d360 0x7ff240152ed0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.683+0000 7ff245645640 1 -- 172.21.15.16:0/1664043661 >> 172.21.15.16:0/1664043661 conn(0x7ff2400affc0 msgr2=0x7ff24015bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.683+0000 7ff245645640 1 -- 172.21.15.16:0/1664043661 shutdown_connections 2024-04-24T23:50:38.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.683+0000 7ff245645640 1 -- 172.21.15.16:0/1664043661 wait complete. 2024-04-24T23:50:38.694 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:50:38.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.797+0000 7f8db7d4f640 1 Processor -- start 2024-04-24T23:50:38.798 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.797+0000 7f8db7d4f640 1 -- start start 2024-04-24T23:50:38.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.798+0000 7f8db7d4f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8db01455f0 0x7f8db006b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.798+0000 7f8db7d4f640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8db006b620 0x7f8db00ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.798+0000 7f8db7d4f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8db015f9c0 con 0x7f8db015d0a0 2024-04-24T23:50:38.799 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.798+0000 7f8db7d4f640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8db01605d0 con 0x7f8db006b620 2024-04-24T23:50:38.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.798+0000 7f8db7d4f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8db015fca0 con 0x7f8db01455f0 2024-04-24T23:50:38.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.798+0000 7f8db52ea640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8db006b620 0x7f8db00ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.798+0000 7f8db62ec640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8db015d0a0 0x7f8db0071a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32886/0 (socket says 172.21.15.16:32886) 2024-04-24T23:50:38.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.798+0000 7f8db62ec640 1 -- 172.21.15.16:0/1666691496 learned_addr learned my addr 172.21.15.16:0/1666691496 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:38.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.799+0000 7f8db5aeb640 1 --2- 172.21.15.16:0/1666691496 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8db01455f0 0x7f8db006b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.799+0000 7f8db52ea640 1 -- 172.21.15.16:0/1666691496 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8db01455f0 msgr2=0x7f8db006b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:38.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.799+0000 7f8db52ea640 1 --2- 172.21.15.16:0/1666691496 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8db01455f0 0x7f8db006b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.799+0000 7f8db52ea640 1 -- 172.21.15.16:0/1666691496 >> v1:172.21.15.16:6789/0 conn(0x7f8db015d0a0 legacy=0x7f8db0071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.801 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.799+0000 7f8db52ea640 1 -- 172.21.15.16:0/1666691496 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db015ff80 con 0x7f8db006b620 2024-04-24T23:50:38.804 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.799+0000 7f8db52ea640 1 --2- 172.21.15.16:0/1666691496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db006b620 0x7f8db00ce760 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f8da00097a0 tx=0x7f8da002a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34cd46d8ce983f30 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.799+0000 7f8db4ae9640 1 -- 172.21.15.16:0/1666691496 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0037070 con 0x7f8db006b620 2024-04-24T23:50:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.799+0000 7f8db4ae9640 1 -- 172.21.15.16:0/1666691496 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8da002ae50 con 0x7f8db006b620 2024-04-24T23:50:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.800+0000 7f8db4ae9640 1 -- 172.21.15.16:0/1666691496 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0032660 con 0x7f8db006b620 2024-04-24T23:50:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.800+0000 7f8db7d4f640 1 -- 172.21.15.16:0/1666691496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db006b620 msgr2=0x7f8db00ce760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.800+0000 7f8db7d4f640 1 --2- 172.21.15.16:0/1666691496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db006b620 0x7f8db00ce760 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f8da00097a0 tx=0x7f8da002a100 comp rx=0 tx=0).stop 2024-04-24T23:50:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.800+0000 7f8db7d4f640 1 -- 172.21.15.16:0/1666691496 shutdown_connections 2024-04-24T23:50:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.800+0000 7f8db7d4f640 1 --2- 172.21.15.16:0/1666691496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db006b620 0x7f8db00ce760 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.800+0000 7f8db7d4f640 1 --2- 172.21.15.16:0/1666691496 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8db01455f0 0x7f8db006b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.800+0000 7f8db7d4f640 1 -- 172.21.15.16:0/1666691496 >> 172.21.15.16:0/1666691496 conn(0x7f8db00cfc50 msgr2=0x7f8db00d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.800+0000 7f8db7d4f640 1 -- 172.21.15.16:0/1666691496 shutdown_connections 2024-04-24T23:50:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.800+0000 7f8db7d4f640 1 -- 172.21.15.16:0/1666691496 wait complete. 2024-04-24T23:50:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.801+0000 7f8db7d4f640 1 Processor -- start 2024-04-24T23:50:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.801+0000 7f8db7d4f640 1 -- start start 2024-04-24T23:50:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db7d4f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8db006b620 0x7f8db013a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db7d4f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8db01455f0 0x7f8db013d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db7d4f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db015d0a0 0x7f8db013f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db7d4f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8db006dcd0 con 0x7f8db015d0a0 2024-04-24T23:50:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db7d4f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8db01459d0 con 0x7f8db006b620 2024-04-24T23:50:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db7d4f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8db006de50 con 0x7f8db01455f0 2024-04-24T23:50:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db62ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db015d0a0 0x7f8db013f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db62ec640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db015d0a0 0x7f8db013f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50856/0 (socket says 172.21.15.16:50856) 2024-04-24T23:50:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db62ec640 1 -- 172.21.15.16:0/2173109023 learned_addr learned my addr 172.21.15.16:0/2173109023 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db52ea640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8db01455f0 0x7f8db013d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db5aeb640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8db006b620 0x7f8db013a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db62ec640 1 -- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8db01455f0 msgr2=0x7f8db013d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db62ec640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8db01455f0 0x7f8db013d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.806 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db62ec640 1 -- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8db006b620 msgr2=0x7f8db013a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db62ec640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8db006b620 0x7f8db013a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db62ec640 1 -- 172.21.15.16:0/2173109023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db00cc550 con 0x7f8db015d0a0 2024-04-24T23:50:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.802+0000 7f8db52ea640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8db01455f0 0x7f8db013d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.803+0000 7f8db62ec640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db015d0a0 0x7f8db013f510 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8dac00caa0 tx=0x7f8dac00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.803+0000 7f8da67fc640 1 -- 172.21.15.16:0/2173109023 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac017070 con 0x7f8db015d0a0 2024-04-24T23:50:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.803+0000 7f8da67fc640 1 -- 172.21.15.16:0/2173109023 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dac00ab40 con 0x7f8db015d0a0 2024-04-24T23:50:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.803+0000 7f8db7d4f640 1 -- 172.21.15.16:0/2173109023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db013ad10 con 0x7f8db015d0a0 2024-04-24T23:50:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.803+0000 7f8da67fc640 1 -- 172.21.15.16:0/2173109023 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dac012400 con 0x7f8db015d0a0 2024-04-24T23:50:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.803+0000 7f8db7d4f640 1 -- 172.21.15.16:0/2173109023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db013de70 con 0x7f8db015d0a0 2024-04-24T23:50:38.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.804+0000 7f8db7d4f640 1 -- 172.21.15.16:0/2173109023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8db0068f90 con 0x7f8db015d0a0 2024-04-24T23:50:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.804+0000 7f8da67fc640 1 -- 172.21.15.16:0/2173109023 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8dac01a070 con 0x7f8db015d0a0 2024-04-24T23:50:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.805+0000 7f8da67fc640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8d9406be30 0x7f8d9406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.805+0000 7f8da67fc640 1 -- 172.21.15.16:0/2173109023 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7f8dac08ab70 con 0x7f8db015d0a0 2024-04-24T23:50:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.805+0000 7f8db5aeb640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8d9406be30 0x7f8d9406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:38.808 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.806+0000 7f8db5aeb640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8d9406be30 0x7f8d9406e2f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8da8006fd0 tx=0x7f8da8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:38.810 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.809+0000 7f8da67fc640 1 -- 172.21.15.16:0/2173109023 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8dac058cd0 con 0x7f8db015d0a0 2024-04-24T23:50:38.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.990+0000 7f8db7d4f640 1 -- 172.21.15.16:0/2173109023 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8db013e210 con 0x7f8db015d0a0 2024-04-24T23:50:38.991 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.991+0000 7f8da67fc640 1 -- 172.21.15.16:0/2173109023 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) ==== 104+0+11 (secure 0 0 0) 0x7f8dac05cfd0 con 0x7f8db015d0a0 2024-04-24T23:50:38.991 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:50:38.993 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.992+0000 7f8db7d4f640 1 -- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8d9406be30 msgr2=0x7f8d9406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.992+0000 7f8db7d4f640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8d9406be30 0x7f8d9406e2f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8da8006fd0 tx=0x7f8da8008040 comp rx=0 tx=0).stop 2024-04-24T23:50:38.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.992+0000 7f8db7d4f640 1 -- 172.21.15.16:0/2173109023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db015d0a0 msgr2=0x7f8db013f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:38.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.992+0000 7f8db7d4f640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db015d0a0 0x7f8db013f510 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f8dac00caa0 tx=0x7f8dac00cf60 comp rx=0 tx=0).stop 2024-04-24T23:50:38.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.992+0000 7f8db7d4f640 1 -- 172.21.15.16:0/2173109023 shutdown_connections 2024-04-24T23:50:38.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.992+0000 7f8db7d4f640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8d9406be30 0x7f8d9406e2f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.992+0000 7f8db7d4f640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8db015d0a0 0x7f8db013f510 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.992+0000 7f8db7d4f640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8db01455f0 0x7f8db013d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.992+0000 7f8db7d4f640 1 --2- 172.21.15.16:0/2173109023 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8db006b620 0x7f8db013a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:38.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.992+0000 7f8db7d4f640 1 -- 172.21.15.16:0/2173109023 >> 172.21.15.16:0/2173109023 conn(0x7f8db00cfc50 msgr2=0x7f8db00b01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:38.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.992+0000 7f8db7d4f640 1 -- 172.21.15.16:0/2173109023 shutdown_connections 2024-04-24T23:50:38.994 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:38.992+0000 7f8db7d4f640 1 -- 172.21.15.16:0/2173109023 wait complete. 2024-04-24T23:50:39.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:50:39.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.108+0000 7f71dd822640 1 Processor -- start 2024-04-24T23:50:39.109 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.108+0000 7f71dd822640 1 -- start start 2024-04-24T23:50:39.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.109+0000 7f71dd822640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f71d806bc40 0x7f71d806c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:39.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.109+0000 7f71dd822640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71d80b7050 0x7f71d80b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:39.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.109+0000 7f71dd822640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71d80d24d0 con 0x7f71d815dfd0 2024-04-24T23:50:39.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.109+0000 7f71dd822640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f71d806c5e0 con 0x7f71d80b7050 2024-04-24T23:50:39.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.109+0000 7f71dd822640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f71d806ba00 con 0x7f71d806bc40 2024-04-24T23:50:39.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.109+0000 7f71d77fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71d80b7050 0x7f71d80b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:39.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.109+0000 7f71d6ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f71d815dfd0 0x7f71d806b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32888/0 (socket says 172.21.15.16:32888) 2024-04-24T23:50:39.111 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.109+0000 7f71d6ffd640 1 -- 172.21.15.16:0/3440504926 learned_addr learned my addr 172.21.15.16:0/3440504926 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:39.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.109+0000 7f71d67fc640 1 --2- 172.21.15.16:0/3440504926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f71d806bc40 0x7f71d806c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:39.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.109+0000 7f71d77fe640 1 -- 172.21.15.16:0/3440504926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f71d806bc40 msgr2=0x7f71d806c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:39.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.109+0000 7f71d77fe640 1 --2- 172.21.15.16:0/3440504926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f71d806bc40 0x7f71d806c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.109+0000 7f71d77fe640 1 -- 172.21.15.16:0/3440504926 >> v1:172.21.15.16:6789/0 conn(0x7f71d815dfd0 legacy=0x7f71d806b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:39.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.109+0000 7f71d77fe640 1 -- 172.21.15.16:0/3440504926 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d80d27b0 con 0x7f71d80b7050 2024-04-24T23:50:39.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.109+0000 7f71d5ffb640 1 -- 172.21.15.16:0/3440504926 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1045667845 0 0) 0x7f71d80d24d0 con 0x7f71d815dfd0 2024-04-24T23:50:39.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.110+0000 7f71d77fe640 1 --2- 172.21.15.16:0/3440504926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71d80b7050 0x7f71d80b74a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009d90 tx=0x7f71c802a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf3a7c433481475b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:39.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.110+0000 7f71d5ffb640 1 -- 172.21.15.16:0/3440504926 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8031600 con 0x7f71d80b7050 2024-04-24T23:50:39.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.110+0000 7f71d5ffb640 1 -- 172.21.15.16:0/3440504926 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c80317a0 con 0x7f71d80b7050 2024-04-24T23:50:39.112 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.110+0000 7f71d5ffb640 1 -- 172.21.15.16:0/3440504926 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8031b70 con 0x7f71d80b7050 2024-04-24T23:50:39.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.111+0000 7f71dd822640 1 -- 172.21.15.16:0/3440504926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71d80b7050 msgr2=0x7f71d80b74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:39.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.111+0000 7f71dd822640 1 --2- 172.21.15.16:0/3440504926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71d80b7050 0x7f71d80b74a0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f71c8009d90 tx=0x7f71c802a5d0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.115 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.111+0000 7f71dd822640 1 -- 172.21.15.16:0/3440504926 shutdown_connections 2024-04-24T23:50:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.111+0000 7f71dd822640 1 --2- 172.21.15.16:0/3440504926 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71d80b7050 0x7f71d80b74a0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.111+0000 7f71dd822640 1 --2- 172.21.15.16:0/3440504926 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f71d806bc40 0x7f71d806c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.111+0000 7f71dd822640 1 -- 172.21.15.16:0/3440504926 >> 172.21.15.16:0/3440504926 conn(0x7f71d80d3dc0 msgr2=0x7f71d80d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.111+0000 7f71dd822640 1 -- 172.21.15.16:0/3440504926 shutdown_connections 2024-04-24T23:50:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.111+0000 7f71dd822640 1 -- 172.21.15.16:0/3440504926 wait complete. 2024-04-24T23:50:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.112+0000 7f71dd822640 1 Processor -- start 2024-04-24T23:50:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.112+0000 7f71dd822640 1 -- start start 2024-04-24T23:50:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71dd822640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f71d806bc40 0x7f71d81f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71dd822640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71d80b7050 0x7f71d80cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71dd822640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71d815dfd0 0x7f71d80ad510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71dd822640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f71d80d3050 con 0x7f71d80b7050 2024-04-24T23:50:39.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71dd822640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f71d80d2d70 con 0x7f71d815dfd0 2024-04-24T23:50:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71dd822640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f71d80cead0 con 0x7f71d806bc40 2024-04-24T23:50:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71d67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71d80b7050 0x7f71d80cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71d67fc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71d80b7050 0x7f71d80cd8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50876/0 (socket says 172.21.15.16:50876) 2024-04-24T23:50:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71d67fc640 1 -- 172.21.15.16:0/278481350 learned_addr learned my addr 172.21.15.16:0/278481350 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71d77fe640 1 --2- 172.21.15.16:0/278481350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71d815dfd0 0x7f71d80ad510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71d6ffd640 1 --2- 172.21.15.16:0/278481350 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f71d806bc40 0x7f71d81f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71d67fc640 1 -- 172.21.15.16:0/278481350 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f71d806bc40 msgr2=0x7f71d81f3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71d67fc640 1 --2- 172.21.15.16:0/278481350 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f71d806bc40 0x7f71d81f3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71d67fc640 1 -- 172.21.15.16:0/278481350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71d815dfd0 msgr2=0x7f71d80ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71d67fc640 1 --2- 172.21.15.16:0/278481350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71d815dfd0 0x7f71d80ad510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.113+0000 7f71d67fc640 1 -- 172.21.15.16:0/278481350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d80aa990 con 0x7f71d80b7050 2024-04-24T23:50:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.114+0000 7f71d67fc640 1 --2- 172.21.15.16:0/278481350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71d80b7050 0x7f71d80cd8c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f71c000d800 tx=0x7f71c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:39.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.114+0000 7f71cffff640 1 -- 172.21.15.16:0/278481350 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c000b6c0 con 0x7f71d80b7050 2024-04-24T23:50:39.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.114+0000 7f71cffff640 1 -- 172.21.15.16:0/278481350 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c0019070 con 0x7f71d80b7050 2024-04-24T23:50:39.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.114+0000 7f71dd822640 1 -- 172.21.15.16:0/278481350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71d80aac70 con 0x7f71d80b7050 2024-04-24T23:50:39.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.114+0000 7f71cffff640 1 -- 172.21.15.16:0/278481350 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c000ba80 con 0x7f71d80b7050 2024-04-24T23:50:39.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.114+0000 7f71dd822640 1 -- 172.21.15.16:0/278481350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71d80ab230 con 0x7f71d80b7050 2024-04-24T23:50:39.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.115+0000 7f71dd822640 1 -- 172.21.15.16:0/278481350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71d8068f90 con 0x7f71d80b7050 2024-04-24T23:50:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.115+0000 7f71cffff640 1 -- 172.21.15.16:0/278481350 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f71c000bc20 con 0x7f71d80b7050 2024-04-24T23:50:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.116+0000 7f71cffff640 1 --2- 172.21.15.16:0/278481350 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f71b806be30 0x7f71b806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.116+0000 7f71cffff640 1 -- 172.21.15.16:0/278481350 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7f71c008b5b0 con 0x7f71d80b7050 2024-04-24T23:50:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.116+0000 7f71d6ffd640 1 --2- 172.21.15.16:0/278481350 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f71b806be30 0x7f71b806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:39.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.117+0000 7f71d6ffd640 1 --2- 172.21.15.16:0/278481350 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f71b806be30 0x7f71b806e2f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f71d0000f60 tx=0x7f71d0007c00 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:39.121 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.120+0000 7f71cffff640 1 -- 172.21.15.16:0/278481350 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f71c0059cc0 con 0x7f71d80b7050 2024-04-24T23:50:39.305 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.305+0000 7f71dd822640 1 -- 172.21.15.16:0/278481350 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f71d80b53c0 con 0x7f71d80b7050 2024-04-24T23:50:39.306 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.305+0000 7f71cffff640 1 -- 172.21.15.16:0/278481350 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) ==== 100+0+11 (secure 0 0 0) 0x7f71c00595c0 con 0x7f71d80b7050 2024-04-24T23:50:39.306 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:50:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.307+0000 7f71dd822640 1 -- 172.21.15.16:0/278481350 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f71b806be30 msgr2=0x7f71b806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.307+0000 7f71dd822640 1 --2- 172.21.15.16:0/278481350 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f71b806be30 0x7f71b806e2f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f71d0000f60 tx=0x7f71d0007c00 comp rx=0 tx=0).stop 2024-04-24T23:50:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.307+0000 7f71dd822640 1 -- 172.21.15.16:0/278481350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71d80b7050 msgr2=0x7f71d80cd8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.307+0000 7f71dd822640 1 --2- 172.21.15.16:0/278481350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71d80b7050 0x7f71d80cd8c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f71c000d800 tx=0x7f71c000dcc0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.307+0000 7f71dd822640 1 -- 172.21.15.16:0/278481350 shutdown_connections 2024-04-24T23:50:39.308 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.307+0000 7f71dd822640 1 --2- 172.21.15.16:0/278481350 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f71b806be30 0x7f71b806e2f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.307+0000 7f71dd822640 1 --2- 172.21.15.16:0/278481350 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71d815dfd0 0x7f71d80ad510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.307+0000 7f71dd822640 1 --2- 172.21.15.16:0/278481350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71d80b7050 0x7f71d80cd8c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.307+0000 7f71dd822640 1 --2- 172.21.15.16:0/278481350 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f71d806bc40 0x7f71d81f3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.307+0000 7f71dd822640 1 -- 172.21.15.16:0/278481350 >> 172.21.15.16:0/278481350 conn(0x7f71d80d3dc0 msgr2=0x7f71d80cf850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.307+0000 7f71dd822640 1 -- 172.21.15.16:0/278481350 shutdown_connections 2024-04-24T23:50:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.307+0000 7f71dd822640 1 -- 172.21.15.16:0/278481350 wait complete. 2024-04-24T23:50:39.319 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T23:50:39.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.420+0000 7ff38e1b7640 1 Processor -- start 2024-04-24T23:50:39.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.420+0000 7ff38e1b7640 1 -- start start 2024-04-24T23:50:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.420+0000 7ff38e1b7640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7ff3880ce7c0 0x7ff3880cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.420+0000 7ff38e1b7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3880cf3a0 con 0x7ff388144740 2024-04-24T23:50:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.420+0000 7ff38e1b7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3880cc8d0 con 0x7ff388071a00 2024-04-24T23:50:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.420+0000 7ff38e1b7640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7ff38815c1f0 con 0x7ff3880ce7c0 2024-04-24T23:50:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff386ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff388144740 0x7ff38815c3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32898/0 (socket says 172.21.15.16:32898) 2024-04-24T23:50:39.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff386ffd640 1 -- 172.21.15.16:0/2342527572 learned_addr learned my addr 172.21.15.16:0/2342527572 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff387fff640 1 --2- 172.21.15.16:0/2342527572 >> v2:172.21.15.156:3300/0 conn(0x7ff3880ce7c0 0x7ff3880cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 -- 172.21.15.16:0/2342527572 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2432086889 0 0) 0x7ff3880cf3a0 con 0x7ff388144740 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 -- 172.21.15.16:0/2342527572 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff370003520 con 0x7ff388144740 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 -- 172.21.15.16:0/2342527572 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3480093521 0 0) 0x7ff370003520 con 0x7ff388144740 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 -- 172.21.15.16:0/2342527572 >> v2:172.21.15.156:3300/0 conn(0x7ff3880ce7c0 msgr2=0x7ff3880cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 --2- 172.21.15.16:0/2342527572 >> v2:172.21.15.156:3300/0 conn(0x7ff3880ce7c0 0x7ff3880cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 -- 172.21.15.16:0/2342527572 >> v1:172.21.15.156:6789/0 conn(0x7ff388071a00 legacy=0x7ff388071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 -- 172.21.15.16:0/2342527572 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3880ccbb0 con 0x7ff388144740 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 -- 172.21.15.16:0/2342527572 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7ff37c001670 con 0x7ff388144740 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 -- 172.21.15.16:0/2342527572 >> v1:172.21.15.16:6789/0 conn(0x7ff388144740 legacy=0x7ff38815c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 --2- 172.21.15.16:0/2342527572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3700039b0 0x7ff370003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 --2- 172.21.15.16:0/2342527572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff370004800 0x7ff370004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 --2- 172.21.15.16:0/2342527572 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3700051c0 0x7ff37000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 -- 172.21.15.16:0/2342527572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff37000c030 con 0x7ff3700039b0 2024-04-24T23:50:39.425 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 -- 172.21.15.16:0/2342527572 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff37000c1e0 con 0x7ff370004800 2024-04-24T23:50:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3867fc640 1 -- 172.21.15.16:0/2342527572 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff37000c3e0 con 0x7ff3700051c0 2024-04-24T23:50:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff3877fe640 1 --2- 172.21.15.16:0/2342527572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3700039b0 0x7ff370003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.421+0000 7ff386ffd640 1 --2- 172.21.15.16:0/2342527572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff370004800 0x7ff370004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.422+0000 7ff387fff640 1 --2- 172.21.15.16:0/2342527572 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3700051c0 0x7ff37000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.422+0000 7ff3877fe640 1 -- 172.21.15.16:0/2342527572 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3700051c0 msgr2=0x7ff37000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.422+0000 7ff3877fe640 1 --2- 172.21.15.16:0/2342527572 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff3700051c0 0x7ff37000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.422+0000 7ff3877fe640 1 -- 172.21.15.16:0/2342527572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff370004800 msgr2=0x7ff370004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.422+0000 7ff3877fe640 1 --2- 172.21.15.16:0/2342527572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff370004800 0x7ff370004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.422+0000 7ff3877fe640 1 -- 172.21.15.16:0/2342527572 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff370003520 con 0x7ff3700039b0 2024-04-24T23:50:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.422+0000 7ff387fff640 1 -- 172.21.15.16:0/2342527572 reap_dead start 2024-04-24T23:50:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.422+0000 7ff3877fe640 1 --2- 172.21.15.16:0/2342527572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3700039b0 0x7ff370003dd0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff378001750 tx=0x7ff370005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c3b5631e64bc86 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.423+0000 7ff3867fc640 1 -- 172.21.15.16:0/2342527572 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff378001370 con 0x7ff3700039b0 2024-04-24T23:50:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.423+0000 7ff3867fc640 1 -- 172.21.15.16:0/2342527572 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3780052d0 con 0x7ff3700039b0 2024-04-24T23:50:39.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.423+0000 7ff3867fc640 1 -- 172.21.15.16:0/2342527572 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37802abc0 con 0x7ff3700039b0 2024-04-24T23:50:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.423+0000 7ff38e1b7640 1 -- 172.21.15.16:0/2342527572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3700039b0 msgr2=0x7ff370003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.423+0000 7ff38e1b7640 1 --2- 172.21.15.16:0/2342527572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3700039b0 0x7ff370003dd0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7ff378001750 tx=0x7ff370005830 comp rx=0 tx=0).stop 2024-04-24T23:50:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.423+0000 7ff38e1b7640 1 -- 172.21.15.16:0/2342527572 shutdown_connections 2024-04-24T23:50:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.423+0000 7ff38e1b7640 1 --2- 172.21.15.16:0/2342527572 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3700039b0 0x7ff370003dd0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.423+0000 7ff38e1b7640 1 -- 172.21.15.16:0/2342527572 >> 172.21.15.16:0/2342527572 conn(0x7ff3880b2000 msgr2=0x7ff3880cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.423+0000 7ff38e1b7640 1 -- 172.21.15.16:0/2342527572 shutdown_connections 2024-04-24T23:50:39.427 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.423+0000 7ff38e1b7640 1 -- 172.21.15.16:0/2342527572 wait complete. 2024-04-24T23:50:39.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.424+0000 7ff38e1b7640 1 Processor -- start 2024-04-24T23:50:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.425+0000 7ff38e1b7640 1 -- start start 2024-04-24T23:50:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.425+0000 7ff38e1b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3700039b0 0x7ff3881f35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.425+0000 7ff38e1b7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff38813f7a0 0x7ff3881ef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.425+0000 7ff38e1b7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3881efae0 0x7ff3880b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.425+0000 7ff38e1b7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff3880cec10 con 0x7ff3700039b0 2024-04-24T23:50:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.425+0000 7ff38e1b7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff3880cd170 con 0x7ff3881efae0 2024-04-24T23:50:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.425+0000 7ff38e1b7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff3880ced90 con 0x7ff38813f7a0 2024-04-24T23:50:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.425+0000 7ff3877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3700039b0 0x7ff3881f35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.425+0000 7ff3877fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3700039b0 0x7ff3881f35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50894/0 (socket says 172.21.15.16:50894) 2024-04-24T23:50:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.425+0000 7ff3877fe640 1 -- 172.21.15.16:0/3130449336 learned_addr learned my addr 172.21.15.16:0/3130449336 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff387fff640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3881efae0 0x7ff3880b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff386ffd640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff38813f7a0 0x7ff3881ef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff3877fe640 1 -- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff38813f7a0 msgr2=0x7ff3881ef5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:39.430 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff3877fe640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff38813f7a0 0x7ff3881ef5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff3877fe640 1 -- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3881efae0 msgr2=0x7ff3880b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff3877fe640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3881efae0 0x7ff3880b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff3877fe640 1 -- 172.21.15.16:0/3130449336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff38813fc20 con 0x7ff3700039b0 2024-04-24T23:50:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff387fff640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3881efae0 0x7ff3880b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff386ffd640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff38813f7a0 0x7ff3881ef5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff3877fe640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3700039b0 0x7ff3881f35c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff378000c00 tx=0x7ff378001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff36ffff640 1 -- 172.21.15.16:0/3130449336 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37803f030 con 0x7ff3700039b0 2024-04-24T23:50:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff36ffff640 1 -- 172.21.15.16:0/3130449336 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff378005480 con 0x7ff3700039b0 2024-04-24T23:50:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff38e1b7640 1 -- 172.21.15.16:0/3130449336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3881efff0 con 0x7ff3700039b0 2024-04-24T23:50:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff36ffff640 1 -- 172.21.15.16:0/3130449336 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff378037040 con 0x7ff3700039b0 2024-04-24T23:50:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.426+0000 7ff38e1b7640 1 -- 172.21.15.16:0/3130449336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3881f2dc0 con 0x7ff3700039b0 2024-04-24T23:50:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.427+0000 7ff38e1b7640 1 -- 172.21.15.16:0/3130449336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3880731f0 con 0x7ff3700039b0 2024-04-24T23:50:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.428+0000 7ff36ffff640 1 -- 172.21.15.16:0/3130449336 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff378042070 con 0x7ff3700039b0 2024-04-24T23:50:39.431 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.428+0000 7ff36ffff640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff37406be60 0x7ff37406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.428+0000 7ff36ffff640 1 -- 172.21.15.16:0/3130449336 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7ff3780aa690 con 0x7ff3700039b0 2024-04-24T23:50:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.428+0000 7ff386ffd640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff37406be60 0x7ff37406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:39.432 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.429+0000 7ff386ffd640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff37406be60 0x7ff37406e320 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff38815cfb0 tx=0x7ff37c009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:39.433 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.432+0000 7ff36ffff640 1 -- 172.21.15.16:0/3130449336 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7ff378078db0 con 0x7ff3700039b0 2024-04-24T23:50:39.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.627+0000 7ff38e1b7640 1 -- 172.21.15.16:0/3130449336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7ff38804f5e0 con 0x7ff3700039b0 2024-04-24T23:50:39.628 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.627+0000 7ff36ffff640 1 -- 172.21.15.16:0/3130449336 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v43) ==== 80+0+120 (secure 0 0 0) 0x7ff3780786b0 con 0x7ff3700039b0 2024-04-24T23:50:39.628 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6834/561619268 2024-04-25T23:50:30.259891+0000 2024-04-24T23:50:39.628 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.16:6832/561619268 2024-04-25T23:50:30.259891+0000 2024-04-24T23:50:39.629 INFO:teuthology.orchestra.run.smithi016.stderr:listed 2 entries 2024-04-24T23:50:39.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.629+0000 7ff38e1b7640 1 -- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff37406be60 msgr2=0x7ff37406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:39.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.629+0000 7ff38e1b7640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff37406be60 0x7ff37406e320 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7ff38815cfb0 tx=0x7ff37c009040 comp rx=0 tx=0).stop 2024-04-24T23:50:39.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.629+0000 7ff38e1b7640 1 -- 172.21.15.16:0/3130449336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3700039b0 msgr2=0x7ff3881f35c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:39.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.629+0000 7ff38e1b7640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3700039b0 0x7ff3881f35c0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7ff378000c00 tx=0x7ff378001120 comp rx=0 tx=0).stop 2024-04-24T23:50:39.630 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.629+0000 7ff38e1b7640 1 -- 172.21.15.16:0/3130449336 shutdown_connections 2024-04-24T23:50:39.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.629+0000 7ff38e1b7640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff37406be60 0x7ff37406e320 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.629+0000 7ff38e1b7640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff3881efae0 0x7ff3880b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.629+0000 7ff38e1b7640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff38813f7a0 0x7ff3881ef5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.629+0000 7ff38e1b7640 1 --2- 172.21.15.16:0/3130449336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff3700039b0 0x7ff3881f35c0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:39.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.629+0000 7ff38e1b7640 1 -- 172.21.15.16:0/3130449336 >> 172.21.15.16:0/3130449336 conn(0x7ff3880b2000 msgr2=0x7ff38806d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:39.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.629+0000 7ff38e1b7640 1 -- 172.21.15.16:0/3130449336 shutdown_connections 2024-04-24T23:50:39.631 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:39.629+0000 7ff38e1b7640 1 -- 172.21.15.16:0/3130449336 wait complete. 2024-04-24T23:50:39.641 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi016.front.sepia.ceph.com... 2024-04-24T23:50:39.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:39.641 DEBUG:teuthology.orchestra.run.smithi016:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-04-24T23:50:39.665 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[51121]: fuse finished with error 0 and tester_r 0 2024-04-24T23:50:39.666 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:50:39.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:39.666 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T23:50:39.722 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T23:50:39.722 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:50:39.722 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:50:39.741 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056108,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1214 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T23:50:39.742 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:50:39.742 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:39.742 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-04-24T23:50:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:50:39.800 INFO:teuthology.orchestra.run.smithi156.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-04-24T23:50:39.800 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T23:50:39.801 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:50:39.801 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:50:39.816 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056882,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19453 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T23:50:39.816 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:50:39.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:39.816 DEBUG:teuthology.orchestra.run.smithi016:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-04-24T23:50:39.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:50:39.872 INFO:teuthology.orchestra.run.smithi016.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-04-24T23:50:39.872 INFO:tasks.cephfs.mount:Terminating background process 2024-04-24T23:50:39.872 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:50:39.872 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:50:39.891 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056108,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1214 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T23:50:39.891 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:50:39.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:39.891 DEBUG:teuthology.orchestra.run.smithi156:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-04-24T23:50:39.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:50:39.954 INFO:teuthology.orchestra.run.smithi156.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-04-24T23:50:39.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:50:40.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.058+0000 7fd1ef961640 1 Processor -- start 2024-04-24T23:50:40.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.059+0000 7fd1ef961640 1 -- start start 2024-04-24T23:50:40.070 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.059+0000 7fd1ef961640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1e815e020 0x7fd1e80cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.059+0000 7fd1ef961640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd1e80b2bb0 0x7fd1e80cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.059+0000 7fd1ef961640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1e80b3390 con 0x7fd1e80cfcf0 2024-04-24T23:50:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.059+0000 7fd1ef961640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd1e80d0100 con 0x7fd1e815e020 2024-04-24T23:50:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.059+0000 7fd1ef961640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fd1e806bfc0 con 0x7fd1e80b2bb0 2024-04-24T23:50:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.060+0000 7fd1ed6fd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1e815e020 0x7fd1e80cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.060+0000 7fd1ed6fd640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd1e815e020 0x7fd1e80cf270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50900/0 (socket says 172.21.15.16:50900) 2024-04-24T23:50:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.060+0000 7fd1ed6fd640 1 -- 172.21.15.16:0/2249360351 learned_addr learned my addr 172.21.15.16:0/2249360351 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.060+0000 7fd1ed6fd640 1 -- 172.21.15.16:0/2249360351 >> v2:172.21.15.156:3300/0 conn(0x7fd1e80b2bb0 msgr2=0x7fd1e80cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.060+0000 7fd1ed6fd640 1 --2- 172.21.15.16:0/2249360351 >> v2:172.21.15.156:3300/0 conn(0x7fd1e80b2bb0 0x7fd1e80cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.060+0000 7fd1ed6fd640 1 -- 172.21.15.16:0/2249360351 >> v1:172.21.15.156:6789/0 conn(0x7fd1e80cfcf0 legacy=0x7fd1e8161bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.060+0000 7fd1ed6fd640 1 -- 172.21.15.16:0/2249360351 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e806c2a0 con 0x7fd1e815e020 2024-04-24T23:50:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.061+0000 7fd1ed6fd640 1 --2- 172.21.15.16:0/2249360351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e815e020 0x7fd1e80cf270 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd1e808d070 tx=0x7fd1d802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=641c195751188714 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:40.073 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.061+0000 7fd1e7fff640 1 -- 172.21.15.16:0/2249360351 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d8036070 con 0x7fd1e815e020 2024-04-24T23:50:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.061+0000 7fd1e7fff640 1 -- 172.21.15.16:0/2249360351 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1d8031600 con 0x7fd1e815e020 2024-04-24T23:50:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.061+0000 7fd1e7fff640 1 -- 172.21.15.16:0/2249360351 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1d80319d0 con 0x7fd1e815e020 2024-04-24T23:50:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.061+0000 7fd1ef961640 1 -- 172.21.15.16:0/2249360351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e815e020 msgr2=0x7fd1e80cf270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.061+0000 7fd1ef961640 1 --2- 172.21.15.16:0/2249360351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e815e020 0x7fd1e80cf270 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd1e808d070 tx=0x7fd1d802a660 comp rx=0 tx=0).stop 2024-04-24T23:50:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.061+0000 7fd1ef961640 1 -- 172.21.15.16:0/2249360351 shutdown_connections 2024-04-24T23:50:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.061+0000 7fd1ef961640 1 --2- 172.21.15.16:0/2249360351 >> v2:172.21.15.156:3300/0 conn(0x7fd1e80b2bb0 0x7fd1e80cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.061+0000 7fd1ef961640 1 --2- 172.21.15.16:0/2249360351 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e815e020 0x7fd1e80cf270 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.061+0000 7fd1ef961640 1 -- 172.21.15.16:0/2249360351 >> 172.21.15.16:0/2249360351 conn(0x7fd1e80d3e10 msgr2=0x7fd1e80d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.061+0000 7fd1ef961640 1 -- 172.21.15.16:0/2249360351 shutdown_connections 2024-04-24T23:50:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.061+0000 7fd1ef961640 1 -- 172.21.15.16:0/2249360351 wait complete. 2024-04-24T23:50:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.062+0000 7fd1ef961640 1 Processor -- start 2024-04-24T23:50:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.063+0000 7fd1ef961640 1 -- start start 2024-04-24T23:50:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.063+0000 7fd1ef961640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd1e80b2bb0 0x7fd1e8154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.074 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.063+0000 7fd1ef961640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e80cfcf0 0x7fd1e80caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.063+0000 7fd1ef961640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1e815e020 0x7fd1e80cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.063+0000 7fd1ef961640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e80d1a80 con 0x7fd1e80cfcf0 2024-04-24T23:50:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.063+0000 7fd1ef961640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e806c860 con 0x7fd1e815e020 2024-04-24T23:50:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.063+0000 7fd1ef961640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fd1e815e400 con 0x7fd1e80b2bb0 2024-04-24T23:50:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.063+0000 7fd1ecefc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e80cfcf0 0x7fd1e80caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.063+0000 7fd1ecefc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e80cfcf0 0x7fd1e80caee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50906/0 (socket says 172.21.15.16:50906) 2024-04-24T23:50:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.063+0000 7fd1ecefc640 1 -- 172.21.15.16:0/174381722 learned_addr learned my addr 172.21.15.16:0/174381722 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1edefe640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1e815e020 0x7fd1e80cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1ecefc640 1 -- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd1e80b2bb0 msgr2=0x7fd1e8154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1ed6fd640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd1e80b2bb0 0x7fd1e8154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1ecefc640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd1e80b2bb0 0x7fd1e8154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1ecefc640 1 -- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1e815e020 msgr2=0x7fd1e80cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1ecefc640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1e815e020 0x7fd1e80cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.075 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1ecefc640 1 -- 172.21.15.16:0/174381722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e80cb8f0 con 0x7fd1e80cfcf0 2024-04-24T23:50:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1edefe640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1e815e020 0x7fd1e80cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1ed6fd640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd1e80b2bb0 0x7fd1e8154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1ecefc640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e80cfcf0 0x7fd1e80caee0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00b440 tx=0x7fd1dc00b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1e5ffb640 1 -- 172.21.15.16:0/174381722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc015020 con 0x7fd1e80cfcf0 2024-04-24T23:50:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1e5ffb640 1 -- 172.21.15.16:0/174381722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1dc011890 con 0x7fd1e80cfcf0 2024-04-24T23:50:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1ef961640 1 -- 172.21.15.16:0/174381722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e80cbbb0 con 0x7fd1e80cfcf0 2024-04-24T23:50:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1e5ffb640 1 -- 172.21.15.16:0/174381722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc011c80 con 0x7fd1e80cfcf0 2024-04-24T23:50:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.064+0000 7fd1ef961640 1 -- 172.21.15.16:0/174381722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e81591a0 con 0x7fd1e80cfcf0 2024-04-24T23:50:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.065+0000 7fd1ef961640 1 -- 172.21.15.16:0/174381722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1e806dfe0 con 0x7fd1e80cfcf0 2024-04-24T23:50:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.066+0000 7fd1e5ffb640 1 -- 172.21.15.16:0/174381722 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd1dc018020 con 0x7fd1e80cfcf0 2024-04-24T23:50:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.066+0000 7fd1e5ffb640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd1d406be30 0x7fd1d406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.066+0000 7fd1e5ffb640 1 -- 172.21.15.16:0/174381722 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7fd1dc08b0e0 con 0x7fd1e80cfcf0 2024-04-24T23:50:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.066+0000 7fd1ed6fd640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd1d406be30 0x7fd1d406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.076 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.067+0000 7fd1ed6fd640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd1d406be30 0x7fd1d406e2f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8033000 tx=0x7fd1d8009000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:40.077 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.070+0000 7fd1e5ffb640 1 -- 172.21.15.16:0/174381722 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fd1dc01f020 con 0x7fd1e80cfcf0 2024-04-24T23:50:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.292+0000 7fd1ef961640 1 -- 172.21.15.16:0/174381722 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1e804f5e0 con 0x7fd1e80cfcf0 2024-04-24T23:50:40.293 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.292+0000 7fd1e5ffb640 1 -- 172.21.15.16:0/174381722 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5184 (secure 0 0 0) 0x7fd1dc00d040 con 0x7fd1e80cfcf0 2024-04-24T23:50:40.294 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:40.294 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5299,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/448535992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":448535992},{"type":"v1","addr":"172.21.15.16:6835","nonce":448535992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5305,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1890687780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1890687780},{"type":"v1","addr":"172.21.15.156:6836","nonce":1890687780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5308,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2538147395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":2538147395},{"type":"v1","addr":"172.21.15.156:6837","nonce":2538147395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:50:31.671778+0000","modified":"2024-04-24T23:50:35.722577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5736},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5736":{"gid":5736,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6834/3643102469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3643102469},{"type":"v1","addr":"172.21.15.16:6834","nonce":3643102469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":5736,"qdb_cluster":[5736]},"id":2}]} 2024-04-24T23:50:40.297 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 23 2024-04-24T23:50:40.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.294+0000 7fd1ef961640 1 -- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd1d406be30 msgr2=0x7fd1d406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.294+0000 7fd1ef961640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd1d406be30 0x7fd1d406e2f0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd1d8033000 tx=0x7fd1d8009000 comp rx=0 tx=0).stop 2024-04-24T23:50:40.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.294+0000 7fd1ef961640 1 -- 172.21.15.16:0/174381722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e80cfcf0 msgr2=0x7fd1e80caee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.294+0000 7fd1ef961640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e80cfcf0 0x7fd1e80caee0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc00b440 tx=0x7fd1dc00b900 comp rx=0 tx=0).stop 2024-04-24T23:50:40.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.295+0000 7fd1ef961640 1 -- 172.21.15.16:0/174381722 shutdown_connections 2024-04-24T23:50:40.297 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.295+0000 7fd1ef961640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd1d406be30 0x7fd1d406e2f0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.295+0000 7fd1ef961640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1e815e020 0x7fd1e80cd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.295+0000 7fd1ef961640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd1e80cfcf0 0x7fd1e80caee0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.298 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.295+0000 7fd1ef961640 1 --2- 172.21.15.16:0/174381722 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd1e80b2bb0 0x7fd1e8154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.295+0000 7fd1ef961640 1 -- 172.21.15.16:0/174381722 >> 172.21.15.16:0/174381722 conn(0x7fd1e80d3e10 msgr2=0x7fd1e80b8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:40.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.295+0000 7fd1ef961640 1 -- 172.21.15.16:0/174381722 shutdown_connections 2024-04-24T23:50:40.299 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.295+0000 7fd1ef961640 1 -- 172.21.15.16:0/174381722 wait complete. 2024-04-24T23:50:40.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:50:40.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.408+0000 7fcccafa5640 1 Processor -- start 2024-04-24T23:50:40.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.408+0000 7fcccafa5640 1 -- start start 2024-04-24T23:50:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.409+0000 7fcccafa5640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fccc4144b20 0x7fccc4144f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.409+0000 7fcccafa5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fccc40ced90 0x7fccc406da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.409+0000 7fcccafa5640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccc4145440 con 0x7fccc41457b0 2024-04-24T23:50:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.409+0000 7fcccafa5640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fccc415fe90 con 0x7fccc4144b20 2024-04-24T23:50:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.409+0000 7fcccafa5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fccc4160170 con 0x7fccc40ced90 2024-04-24T23:50:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.409+0000 7fccc9542640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fccc40ced90 0x7fccc406da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.409+0000 7fccc8d41640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fccc4144b20 0x7fccc4144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.409+0000 7fccc9542640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fccc40ced90 0x7fccc406da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:55772/0 (socket says 172.21.15.16:55772) 2024-04-24T23:50:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.409+0000 7fccc9542640 1 -- 172.21.15.16:0/2961317723 learned_addr learned my addr 172.21.15.16:0/2961317723 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.409+0000 7fccc3fff640 1 --1- 172.21.15.16:0/2961317723 >> v1:172.21.15.156:6789/0 conn(0x7fccc41457b0 0x7fccc40ce680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:53504/0 (socket says 172.21.15.16:53504) 2024-04-24T23:50:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.410+0000 7fccc9542640 1 -- 172.21.15.16:0/2961317723 >> v2:172.21.15.156:3300/0 conn(0x7fccc4144b20 msgr2=0x7fccc4144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:40.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.410+0000 7fccc9542640 1 --2- 172.21.15.16:0/2961317723 >> v2:172.21.15.156:3300/0 conn(0x7fccc4144b20 0x7fccc4144f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.410+0000 7fccc9542640 1 -- 172.21.15.16:0/2961317723 >> v1:172.21.15.156:6789/0 conn(0x7fccc41457b0 legacy=0x7fccc40ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.410+0000 7fccc9542640 1 -- 172.21.15.16:0/2961317723 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc4160450 con 0x7fccc40ced90 2024-04-24T23:50:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.410+0000 7fccc8d41640 1 --2- 172.21.15.16:0/2961317723 >> v2:172.21.15.156:3300/0 conn(0x7fccc4144b20 0x7fccc4144f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:50:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.410+0000 7fccc9542640 1 --2- 172.21.15.16:0/2961317723 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fccc40ced90 0x7fccc406da40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009e50 tx=0x7fccbc02a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8075f7137fc1964 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.411+0000 7fccc37fe640 1 -- 172.21.15.16:0/2961317723 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc036070 con 0x7fccc40ced90 2024-04-24T23:50:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.411+0000 7fccc37fe640 1 -- 172.21.15.16:0/2961317723 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccbc031600 con 0x7fccc40ced90 2024-04-24T23:50:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.411+0000 7fccc37fe640 1 -- 172.21.15.16:0/2961317723 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc0319d0 con 0x7fccc40ced90 2024-04-24T23:50:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.411+0000 7fcccafa5640 1 -- 172.21.15.16:0/2961317723 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fccc40ced90 msgr2=0x7fccc406da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.411+0000 7fcccafa5640 1 --2- 172.21.15.16:0/2961317723 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fccc40ced90 0x7fccc406da40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009e50 tx=0x7fccbc02a680 comp rx=0 tx=0).stop 2024-04-24T23:50:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.411+0000 7fcccafa5640 1 -- 172.21.15.16:0/2961317723 shutdown_connections 2024-04-24T23:50:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.411+0000 7fcccafa5640 1 --2- 172.21.15.16:0/2961317723 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fccc40ced90 0x7fccc406da40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.411+0000 7fcccafa5640 1 --2- 172.21.15.16:0/2961317723 >> v2:172.21.15.156:3300/0 conn(0x7fccc4144b20 0x7fccc4144f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.411+0000 7fcccafa5640 1 -- 172.21.15.16:0/2961317723 >> 172.21.15.16:0/2961317723 conn(0x7fccc40b2000 msgr2=0x7fccc40cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:40.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.411+0000 7fcccafa5640 1 -- 172.21.15.16:0/2961317723 shutdown_connections 2024-04-24T23:50:40.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.411+0000 7fcccafa5640 1 -- 172.21.15.16:0/2961317723 wait complete. 2024-04-24T23:50:40.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.412+0000 7fcccafa5640 1 Processor -- start 2024-04-24T23:50:40.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.412+0000 7fcccafa5640 1 -- start start 2024-04-24T23:50:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.413+0000 7fcccafa5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fccc40ced90 0x7fccc413e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.413+0000 7fcccafa5640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fccc4144b20 0x7fccc41f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.413+0000 7fcccafa5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc41457b0 0x7fccc415e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.413+0000 7fcccafa5640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fccc40b3390 con 0x7fccc41457b0 2024-04-24T23:50:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.413+0000 7fcccafa5640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fccc406d4b0 con 0x7fccc4144b20 2024-04-24T23:50:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.413+0000 7fcccafa5640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fccc415f990 con 0x7fccc40ced90 2024-04-24T23:50:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.413+0000 7fccc9542640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc41457b0 0x7fccc415e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.413+0000 7fccc9542640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc41457b0 0x7fccc415e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50912/0 (socket says 172.21.15.16:50912) 2024-04-24T23:50:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.413+0000 7fccc9542640 1 -- 172.21.15.16:0/3550271375 learned_addr learned my addr 172.21.15.16:0/3550271375 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fccc8d41640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fccc40ced90 0x7fccc413e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fccc3fff640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fccc4144b20 0x7fccc41f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fccc9542640 1 -- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fccc40ced90 msgr2=0x7fccc413e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fccc9542640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fccc40ced90 0x7fccc413e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fccc9542640 1 -- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fccc4144b20 msgr2=0x7fccc41f5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fccc9542640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fccc4144b20 0x7fccc41f5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fccc9542640 1 -- 172.21.15.16:0/3550271375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccc415f190 con 0x7fccc41457b0 2024-04-24T23:50:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fccc8d41640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fccc40ced90 0x7fccc413e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fccc3fff640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fccc4144b20 0x7fccc41f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fccc9542640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc41457b0 0x7fccc415e970 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009f70 tx=0x7fccbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fccc17fa640 1 -- 172.21.15.16:0/3550271375 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc03f020 con 0x7fccc41457b0 2024-04-24T23:50:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fccc17fa640 1 -- 172.21.15.16:0/3550271375 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccbc037040 con 0x7fccc41457b0 2024-04-24T23:50:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fcccafa5640 1 -- 172.21.15.16:0/3550271375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccc41f1e10 con 0x7fccc41457b0 2024-04-24T23:50:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fccc17fa640 1 -- 172.21.15.16:0/3550271375 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccbc036070 con 0x7fccc41457b0 2024-04-24T23:50:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.414+0000 7fcccafa5640 1 -- 172.21.15.16:0/3550271375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccc41f23d0 con 0x7fccc41457b0 2024-04-24T23:50:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.415+0000 7fcccafa5640 1 -- 172.21.15.16:0/3550271375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccc40731f0 con 0x7fccc41457b0 2024-04-24T23:50:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.416+0000 7fccc17fa640 1 -- 172.21.15.16:0/3550271375 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fccbc042020 con 0x7fccc41457b0 2024-04-24T23:50:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.416+0000 7fccc17fa640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcca806be30 0x7fcca806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.416+0000 7fccc17fa640 1 -- 172.21.15.16:0/3550271375 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7fccbc0ab4d0 con 0x7fccc41457b0 2024-04-24T23:50:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.416+0000 7fccc8d41640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcca806be30 0x7fcca806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.417+0000 7fccc8d41640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcca806be30 0x7fcca806e2f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009ac0 tx=0x7fccb8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:40.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.420+0000 7fccc17fa640 1 -- 172.21.15.16:0/3550271375 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fccbc049030 con 0x7fccc41457b0 2024-04-24T23:50:40.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.607+0000 7fcccafa5640 1 -- 172.21.15.16:0/3550271375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fccc404f5e0 con 0x7fccc41457b0 2024-04-24T23:50:40.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.608+0000 7fccc17fa640 1 -- 172.21.15.16:0/3550271375 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+17050 (secure 0 0 0) 0x7fccbc079c80 con 0x7fccc41457b0 2024-04-24T23:50:40.609 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:40.609 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":43,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:50:37.287284+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:50:17.026573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-24T23:50:17.415963+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.36","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6834/561619268":"2024-04-25T23:50:30.259891+0000","172.21.15.16:6832/561619268":"2024-04-25T23:50:30.259891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:50:40.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.610+0000 7fcccafa5640 1 -- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcca806be30 msgr2=0x7fcca806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.610+0000 7fcccafa5640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcca806be30 0x7fcca806e2f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fccb8009ac0 tx=0x7fccb8008040 comp rx=0 tx=0).stop 2024-04-24T23:50:40.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.610+0000 7fcccafa5640 1 -- 172.21.15.16:0/3550271375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc41457b0 msgr2=0x7fccc415e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.610+0000 7fcccafa5640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc41457b0 0x7fccc415e970 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fccbc009f70 tx=0x7fccbc004a40 comp rx=0 tx=0).stop 2024-04-24T23:50:40.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.610+0000 7fcccafa5640 1 -- 172.21.15.16:0/3550271375 shutdown_connections 2024-04-24T23:50:40.611 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.610+0000 7fcccafa5640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fcca806be30 0x7fcca806e2f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.610+0000 7fcccafa5640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fccc41457b0 0x7fccc415e970 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.610+0000 7fcccafa5640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fccc4144b20 0x7fccc41f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.610+0000 7fcccafa5640 1 --2- 172.21.15.16:0/3550271375 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fccc40ced90 0x7fccc413e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.610+0000 7fcccafa5640 1 -- 172.21.15.16:0/3550271375 >> 172.21.15.16:0/3550271375 conn(0x7fccc40b2000 msgr2=0x7fccc40b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:40.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.610+0000 7fcccafa5640 1 -- 172.21.15.16:0/3550271375 shutdown_connections 2024-04-24T23:50:40.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.610+0000 7fcccafa5640 1 -- 172.21.15.16:0/3550271375 wait complete. 2024-04-24T23:50:40.622 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T23:48:43.160738+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T23:50:17.026573+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T23:50:17.415963+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T23:50:40.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T23:50:40.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.726+0000 7f34fe009640 1 Processor -- start 2024-04-24T23:50:40.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.726+0000 7f34fe009640 1 -- start start 2024-04-24T23:50:40.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.727+0000 7f34fe009640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f34f815d0a0 0x7f34f80ce760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.727+0000 7f34fe009640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34f80ceca0 0x7f34f806ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.727+0000 7f34fe009640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34f815f9c0 con 0x7f34f81455f0 2024-04-24T23:50:40.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.727+0000 7f34fe009640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f34f80cf100 con 0x7f34f80ceca0 2024-04-24T23:50:40.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.727+0000 7f34fe009640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f34f81605d0 con 0x7f34f815d0a0 2024-04-24T23:50:40.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.727+0000 7f34f7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34f80ceca0 0x7f34f806ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.727+0000 7f34f7fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f34f80ceca0 0x7f34f806ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50914/0 (socket says 172.21.15.16:50914) 2024-04-24T23:50:40.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.727+0000 7f34f7fff640 1 -- 172.21.15.16:0/709156596 learned_addr learned my addr 172.21.15.16:0/709156596 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:40.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.727+0000 7f34f6ffd640 1 --2- 172.21.15.16:0/709156596 >> v2:172.21.15.156:3300/0 conn(0x7f34f815d0a0 0x7f34f80ce760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.727+0000 7f34f7fff640 1 -- 172.21.15.16:0/709156596 >> v2:172.21.15.156:3300/0 conn(0x7f34f815d0a0 msgr2=0x7f34f80ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:40.728 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.727+0000 7f34f7fff640 1 --2- 172.21.15.16:0/709156596 >> v2:172.21.15.156:3300/0 conn(0x7f34f815d0a0 0x7f34f80ce760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.727+0000 7f34f7fff640 1 -- 172.21.15.16:0/709156596 >> v1:172.21.15.156:6789/0 conn(0x7f34f81455f0 legacy=0x7f34f806b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:40.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.727+0000 7f34f7fff640 1 -- 172.21.15.16:0/709156596 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f815fca0 con 0x7f34f80ceca0 2024-04-24T23:50:40.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.728+0000 7f34f7fff640 1 --2- 172.21.15.16:0/709156596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f80ceca0 0x7f34f806ccc0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f34f80cf340 tx=0x7f34f002a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=897ed323a4b70eb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:40.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.728+0000 7f34f67fc640 1 -- 172.21.15.16:0/709156596 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f0036070 con 0x7f34f80ceca0 2024-04-24T23:50:40.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.728+0000 7f34f67fc640 1 -- 172.21.15.16:0/709156596 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34f0031600 con 0x7f34f80ceca0 2024-04-24T23:50:40.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.728+0000 7f34f67fc640 1 -- 172.21.15.16:0/709156596 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34f00319d0 con 0x7f34f80ceca0 2024-04-24T23:50:40.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.729+0000 7f34fe009640 1 -- 172.21.15.16:0/709156596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f80ceca0 msgr2=0x7f34f806ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.729+0000 7f34fe009640 1 --2- 172.21.15.16:0/709156596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f80ceca0 0x7f34f806ccc0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f34f80cf340 tx=0x7f34f002a660 comp rx=0 tx=0).stop 2024-04-24T23:50:40.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.729+0000 7f34fe009640 1 -- 172.21.15.16:0/709156596 shutdown_connections 2024-04-24T23:50:40.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.729+0000 7f34fe009640 1 --2- 172.21.15.16:0/709156596 >> v2:172.21.15.156:3300/0 conn(0x7f34f815d0a0 0x7f34f80ce760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.729+0000 7f34fe009640 1 --2- 172.21.15.16:0/709156596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f80ceca0 0x7f34f806ccc0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.729+0000 7f34fe009640 1 -- 172.21.15.16:0/709156596 >> 172.21.15.16:0/709156596 conn(0x7f34f80cfc50 msgr2=0x7f34f80d0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:40.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.729+0000 7f34fe009640 1 -- 172.21.15.16:0/709156596 shutdown_connections 2024-04-24T23:50:40.731 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.729+0000 7f34fe009640 1 -- 172.21.15.16:0/709156596 wait complete. 2024-04-24T23:50:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.730+0000 7f34fe009640 1 Processor -- start 2024-04-24T23:50:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.730+0000 7f34fe009640 1 -- start start 2024-04-24T23:50:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34fe009640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f80ceca0 0x7f34f815af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34fe009640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34f81455f0 0x7f34f8143080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34fe009640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34f815d0a0 0x7f34f81435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34fe009640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f34f806b7f0 con 0x7f34f80ceca0 2024-04-24T23:50:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34fe009640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f34f8160260 con 0x7f34f81455f0 2024-04-24T23:50:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34fe009640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f34f81459d0 con 0x7f34f815d0a0 2024-04-24T23:50:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34f77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f80ceca0 0x7f34f815af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34f77fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f80ceca0 0x7f34f815af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50918/0 (socket says 172.21.15.16:50918) 2024-04-24T23:50:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34f77fe640 1 -- 172.21.15.16:0/921885527 learned_addr learned my addr 172.21.15.16:0/921885527 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34f6ffd640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34f81455f0 0x7f34f8143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34f7fff640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34f815d0a0 0x7f34f81435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34f77fe640 1 -- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34f815d0a0 msgr2=0x7f34f81435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34f77fe640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34f815d0a0 0x7f34f81435c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34f77fe640 1 -- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34f81455f0 msgr2=0x7f34f8143080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34f77fe640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34f81455f0 0x7f34f8143080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34f77fe640 1 -- 172.21.15.16:0/921885527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34f815b9d0 con 0x7f34f80ceca0 2024-04-24T23:50:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34f7fff640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34f815d0a0 0x7f34f81435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.731+0000 7f34f77fe640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f80ceca0 0x7f34f815af80 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f34e800c410 tx=0x7f34e800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.732+0000 7f34dffff640 1 -- 172.21.15.16:0/921885527 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e8017070 con 0x7f34f80ceca0 2024-04-24T23:50:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.732+0000 7f34dffff640 1 -- 172.21.15.16:0/921885527 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34e80035f0 con 0x7f34f80ceca0 2024-04-24T23:50:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.732+0000 7f34fe009640 1 -- 172.21.15.16:0/921885527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34f813af20 con 0x7f34f80ceca0 2024-04-24T23:50:40.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.732+0000 7f34dffff640 1 -- 172.21.15.16:0/921885527 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e8003a20 con 0x7f34f80ceca0 2024-04-24T23:50:40.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.732+0000 7f34fe009640 1 -- 172.21.15.16:0/921885527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34f813b4e0 con 0x7f34f80ceca0 2024-04-24T23:50:40.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.733+0000 7f34fe009640 1 -- 172.21.15.16:0/921885527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34f8068f90 con 0x7f34f80ceca0 2024-04-24T23:50:40.735 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.733+0000 7f34dffff640 1 -- 172.21.15.16:0/921885527 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f34e801a070 con 0x7f34f80ceca0 2024-04-24T23:50:40.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.733+0000 7f34dffff640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f34e406be30 0x7f34e406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:40.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.734+0000 7f34dffff640 1 -- 172.21.15.16:0/921885527 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7f34e808a480 con 0x7f34f80ceca0 2024-04-24T23:50:40.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.734+0000 7f34f6ffd640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f34e406be30 0x7f34e406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:40.736 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.734+0000 7f34f6ffd640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f34e406be30 0x7f34e406e2f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f34f80b4820 tx=0x7f34ec008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:40.738 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.738+0000 7f34dffff640 1 -- 172.21.15.16:0/921885527 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f34e8058cc0 con 0x7f34f80ceca0 2024-04-24T23:50:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.919+0000 7f34fe009640 1 -- 172.21.15.16:0/921885527 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f34f80b3150 con 0x7f34f80ceca0 2024-04-24T23:50:40.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.920+0000 7f34dffff640 1 -- 172.21.15.16:0/921885527 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) ==== 93+0+10 (secure 0 0 0) 0x7f34e80010c0 con 0x7f34f80ceca0 2024-04-24T23:50:40.920 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-04-24T23:50:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.922+0000 7f34fe009640 1 -- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f34e406be30 msgr2=0x7f34e406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.922+0000 7f34fe009640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f34e406be30 0x7f34e406e2f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f34f80b4820 tx=0x7f34ec008040 comp rx=0 tx=0).stop 2024-04-24T23:50:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.922+0000 7f34fe009640 1 -- 172.21.15.16:0/921885527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f80ceca0 msgr2=0x7f34f815af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.922+0000 7f34fe009640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f80ceca0 0x7f34f815af80 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f34e800c410 tx=0x7f34e800c8d0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.922+0000 7f34fe009640 1 -- 172.21.15.16:0/921885527 shutdown_connections 2024-04-24T23:50:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.922+0000 7f34fe009640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f34e406be30 0x7f34e406e2f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.922+0000 7f34fe009640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f34f815d0a0 0x7f34f81435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.922+0000 7f34fe009640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34f81455f0 0x7f34f8143080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.922+0000 7f34fe009640 1 --2- 172.21.15.16:0/921885527 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f34f80ceca0 0x7f34f815af80 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.922+0000 7f34fe009640 1 -- 172.21.15.16:0/921885527 >> 172.21.15.16:0/921885527 conn(0x7f34f80cfc50 msgr2=0x7f34f80b0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.922+0000 7f34fe009640 1 -- 172.21.15.16:0/921885527 shutdown_connections 2024-04-24T23:50:40.923 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:40.922+0000 7f34fe009640 1 -- 172.21.15.16:0/921885527 wait complete. 2024-04-24T23:50:40.933 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T23:50:41.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.037+0000 7fbd3697e640 1 Processor -- start 2024-04-24T23:50:41.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.037+0000 7fbd3697e640 1 -- start start 2024-04-24T23:50:41.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd3697e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3006c080 0x7fbd300ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd3697e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd300b66c0 con 0x7fbd30161510 2024-04-24T23:50:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd3697e640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd300aebb0 con 0x7fbd300b6230 2024-04-24T23:50:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd3697e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fbd300adfa0 con 0x7fbd3006c080 2024-04-24T23:50:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbd30161510 0x7fbd3006b970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32912/0 (socket says 172.21.15.16:32912) 2024-04-24T23:50:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2ffff640 1 -- 172.21.15.16:0/2164440325 learned_addr learned my addr 172.21.15.16:0/2164440325 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 -- 172.21.15.16:0/2164440325 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1659780241 0 0) 0x7fbd20001110 con 0x7fbd30161510 2024-04-24T23:50:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 -- 172.21.15.16:0/2164440325 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbd18003520 con 0x7fbd30161510 2024-04-24T23:50:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2f7fe640 1 --2- 172.21.15.16:0/2164440325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3006c080 0x7fbd300ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 -- 172.21.15.16:0/2164440325 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2828447356 0 0) 0x7fbd18003520 con 0x7fbd30161510 2024-04-24T23:50:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 -- 172.21.15.16:0/2164440325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3006c080 msgr2=0x7fbd300ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 --2- 172.21.15.16:0/2164440325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3006c080 0x7fbd300ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 -- 172.21.15.16:0/2164440325 >> v1:172.21.15.156:6789/0 conn(0x7fbd300b6230 legacy=0x7fbd300b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 -- 172.21.15.16:0/2164440325 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd300ae280 con 0x7fbd30161510 2024-04-24T23:50:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 -- 172.21.15.16:0/2164440325 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7fbd20001840 con 0x7fbd30161510 2024-04-24T23:50:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2f7fe640 1 --2- 172.21.15.16:0/2164440325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3006c080 0x7fbd300ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 -- 172.21.15.16:0/2164440325 >> v1:172.21.15.16:6789/0 conn(0x7fbd30161510 legacy=0x7fbd3006b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 --2- 172.21.15.16:0/2164440325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd180039e0 0x7fbd18003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.045 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 --2- 172.21.15.16:0/2164440325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd18004800 0x7fbd18004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 --2- 172.21.15.16:0/2164440325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3006c080 0x7fbd180051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 -- 172.21.15.16:0/2164440325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbd300adfa0 con 0x7fbd180039e0 2024-04-24T23:50:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 -- 172.21.15.16:0/2164440325 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbd1800bac0 con 0x7fbd18004800 2024-04-24T23:50:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.038+0000 7fbd2effd640 1 -- 172.21.15.16:0/2164440325 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fbd1800bcc0 con 0x7fbd3006c080 2024-04-24T23:50:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.039+0000 7fbd2ffff640 1 --2- 172.21.15.16:0/2164440325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd180039e0 0x7fbd18003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.039+0000 7fbd2f7fe640 1 --2- 172.21.15.16:0/2164440325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd18004800 0x7fbd18004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.039+0000 7fbd34f1b640 1 --2- 172.21.15.16:0/2164440325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3006c080 0x7fbd180051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.039+0000 7fbd2ffff640 1 -- 172.21.15.16:0/2164440325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3006c080 msgr2=0x7fbd180051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.039+0000 7fbd2ffff640 1 --2- 172.21.15.16:0/2164440325 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3006c080 0x7fbd180051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.039+0000 7fbd2ffff640 1 -- 172.21.15.16:0/2164440325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd18004800 msgr2=0x7fbd18004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.039+0000 7fbd2ffff640 1 --2- 172.21.15.16:0/2164440325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd18004800 0x7fbd18004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.039+0000 7fbd2ffff640 1 -- 172.21.15.16:0/2164440325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd18003520 con 0x7fbd180039e0 2024-04-24T23:50:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.039+0000 7fbd2f7fe640 1 --2- 172.21.15.16:0/2164440325 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd18004800 0x7fbd18004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:41.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.039+0000 7fbd2ffff640 1 --2- 172.21.15.16:0/2164440325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd180039e0 0x7fbd18003e00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fbd2000c4c0 tx=0x7fbd18005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34145f9efad2b9ae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.040+0000 7fbd2effd640 1 -- 172.21.15.16:0/2164440325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd2000c0e0 con 0x7fbd180039e0 2024-04-24T23:50:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.040+0000 7fbd2effd640 1 -- 172.21.15.16:0/2164440325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd20034070 con 0x7fbd180039e0 2024-04-24T23:50:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.040+0000 7fbd2effd640 1 -- 172.21.15.16:0/2164440325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20005d30 con 0x7fbd180039e0 2024-04-24T23:50:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.040+0000 7fbd3697e640 1 -- 172.21.15.16:0/2164440325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd180039e0 msgr2=0x7fbd18003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.040+0000 7fbd3697e640 1 --2- 172.21.15.16:0/2164440325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd180039e0 0x7fbd18003e00 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fbd2000c4c0 tx=0x7fbd18005830 comp rx=0 tx=0).stop 2024-04-24T23:50:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.040+0000 7fbd34f1b640 1 -- 172.21.15.16:0/2164440325 reap_dead start 2024-04-24T23:50:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.040+0000 7fbd3697e640 1 -- 172.21.15.16:0/2164440325 shutdown_connections 2024-04-24T23:50:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.040+0000 7fbd3697e640 1 -- 172.21.15.16:0/2164440325 >> 172.21.15.16:0/2164440325 conn(0x7fbd30072aa0 msgr2=0x7fbd30072eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.040+0000 7fbd3697e640 1 -- 172.21.15.16:0/2164440325 shutdown_connections 2024-04-24T23:50:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.040+0000 7fbd3697e640 1 -- 172.21.15.16:0/2164440325 wait complete. 2024-04-24T23:50:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.041+0000 7fbd3697e640 1 Processor -- start 2024-04-24T23:50:41.047 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.042+0000 7fbd3697e640 1 -- start start 2024-04-24T23:50:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.042+0000 7fbd3697e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd180039e0 0x7fbd30160050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.042+0000 7fbd3697e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd30145850 0x7fbd3015e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.042+0000 7fbd3697e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3015ae10 0x7fbd3015b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.042+0000 7fbd3697e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbd3006da90 con 0x7fbd180039e0 2024-04-24T23:50:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.042+0000 7fbd3697e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbd300ae840 con 0x7fbd30145850 2024-04-24T23:50:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.042+0000 7fbd3697e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fbd3006dc10 con 0x7fbd3015ae10 2024-04-24T23:50:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.042+0000 7fbd2ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd180039e0 0x7fbd30160050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.050 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.042+0000 7fbd2ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd180039e0 0x7fbd30160050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50938/0 (socket says 172.21.15.16:50938) 2024-04-24T23:50:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.042+0000 7fbd2ffff640 1 -- 172.21.15.16:0/3933951445 learned_addr learned my addr 172.21.15.16:0/3933951445 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd2f7fe640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd30145850 0x7fbd3015e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd34f1b640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3015ae10 0x7fbd3015b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd2ffff640 1 -- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3015ae10 msgr2=0x7fbd3015b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd2ffff640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3015ae10 0x7fbd3015b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd2ffff640 1 -- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd30145850 msgr2=0x7fbd3015e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd2ffff640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd30145850 0x7fbd3015e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd2ffff640 1 -- 172.21.15.16:0/3933951445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd30158be0 con 0x7fbd180039e0 2024-04-24T23:50:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd2f7fe640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd30145850 0x7fbd3015e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd34f1b640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3015ae10 0x7fbd3015b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd2ffff640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd180039e0 0x7fbd30160050 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fbd20001b40 tx=0x7fbd20001c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd2cff9640 1 -- 172.21.15.16:0/3933951445 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20007480 con 0x7fbd180039e0 2024-04-24T23:50:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd2cff9640 1 -- 172.21.15.16:0/3933951445 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd20000ed0 con 0x7fbd180039e0 2024-04-24T23:50:41.051 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd2cff9640 1 -- 172.21.15.16:0/3933951445 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd20001110 con 0x7fbd180039e0 2024-04-24T23:50:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd3697e640 1 -- 172.21.15.16:0/3933951445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd30160730 con 0x7fbd180039e0 2024-04-24T23:50:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.043+0000 7fbd3697e640 1 -- 172.21.15.16:0/3933951445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd30160cf0 con 0x7fbd180039e0 2024-04-24T23:50:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.044+0000 7fbd3697e640 1 -- 172.21.15.16:0/3933951445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd30068f90 con 0x7fbd180039e0 2024-04-24T23:50:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.045+0000 7fbd2cff9640 1 -- 172.21.15.16:0/3933951445 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbd20039480 con 0x7fbd180039e0 2024-04-24T23:50:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.045+0000 7fbd2cff9640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbd1c06be60 0x7fbd1c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.045+0000 7fbd2cff9640 1 -- 172.21.15.16:0/3933951445 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7fbd200ac3b0 con 0x7fbd180039e0 2024-04-24T23:50:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.045+0000 7fbd2f7fe640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbd1c06be60 0x7fbd1c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.046+0000 7fbd2f7fe640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbd1c06be60 0x7fbd1c06e320 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbd3015c550 tx=0x7fbd24008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.050+0000 7fbd2cff9640 1 -- 172.21.15.16:0/3933951445 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbd300b66c0 con 0x7fbd180039e0 2024-04-24T23:50:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.232+0000 7fbd3697e640 1 -- 172.21.15.16:0/3933951445 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbd3006c860 con 0x7fbd180039e0 2024-04-24T23:50:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.232+0000 7fbd2cff9640 1 -- 172.21.15.16:0/3933951445 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v43) ==== 104+0+11 (secure 0 0 0) 0x7fbd2007aad0 con 0x7fbd180039e0 2024-04-24T23:50:41.233 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 32 2024-04-24T23:50:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.235+0000 7fbd3697e640 1 -- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbd1c06be60 msgr2=0x7fbd1c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.235+0000 7fbd3697e640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbd1c06be60 0x7fbd1c06e320 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbd3015c550 tx=0x7fbd24008040 comp rx=0 tx=0).stop 2024-04-24T23:50:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.235+0000 7fbd3697e640 1 -- 172.21.15.16:0/3933951445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd180039e0 msgr2=0x7fbd30160050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.235+0000 7fbd3697e640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd180039e0 0x7fbd30160050 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fbd20001b40 tx=0x7fbd20001c10 comp rx=0 tx=0).stop 2024-04-24T23:50:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.235+0000 7fbd3697e640 1 -- 172.21.15.16:0/3933951445 shutdown_connections 2024-04-24T23:50:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.235+0000 7fbd3697e640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fbd1c06be60 0x7fbd1c06e320 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.235+0000 7fbd3697e640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbd3015ae10 0x7fbd3015b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.235+0000 7fbd3697e640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd30145850 0x7fbd3015e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.235+0000 7fbd3697e640 1 --2- 172.21.15.16:0/3933951445 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd180039e0 0x7fbd30160050 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.235+0000 7fbd3697e640 1 -- 172.21.15.16:0/3933951445 >> 172.21.15.16:0/3933951445 conn(0x7fbd30072aa0 msgr2=0x7fbd300b13d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:41.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.235+0000 7fbd3697e640 1 -- 172.21.15.16:0/3933951445 shutdown_connections 2024-04-24T23:50:41.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.235+0000 7fbd3697e640 1 -- 172.21.15.16:0/3933951445 wait complete. 2024-04-24T23:50:41.246 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T23:50:41.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.349+0000 7fcd18e38640 1 Processor -- start 2024-04-24T23:50:41.350 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.349+0000 7fcd18e38640 1 -- start start 2024-04-24T23:50:41.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.350+0000 7fcd18e38640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd1415dfd0 0x7fcd1406b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.350+0000 7fcd18e38640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fcd1406bc40 0x7fcd1406c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.350+0000 7fcd18e38640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd140cefc0 0x7fcd140cf410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.350+0000 7fcd18e38640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fcd1406c5e0 con 0x7fcd140cefc0 2024-04-24T23:50:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.350+0000 7fcd18e38640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fcd140d24d0 con 0x7fcd1406bc40 2024-04-24T23:50:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.350+0000 7fcd18e38640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fcd140d27b0 con 0x7fcd1415dfd0 2024-04-24T23:50:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.350+0000 7fcd12d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd140cefc0 0x7fcd140cf410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.350+0000 7fcd12d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd140cefc0 0x7fcd140cf410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50950/0 (socket says 172.21.15.16:50950) 2024-04-24T23:50:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.350+0000 7fcd12d9d640 1 -- 172.21.15.16:0/4247686327 learned_addr learned my addr 172.21.15.16:0/4247686327 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.351+0000 7fcd11d9b640 1 --2- 172.21.15.16:0/4247686327 >> v2:172.21.15.156:3300/0 conn(0x7fcd1406bc40 0x7fcd1406c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.351+0000 7fcd1259c640 1 --2- 172.21.15.16:0/4247686327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd1415dfd0 0x7fcd1406b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.351+0000 7fcd12d9d640 1 -- 172.21.15.16:0/4247686327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd1415dfd0 msgr2=0x7fcd1406b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.351+0000 7fcd12d9d640 1 --2- 172.21.15.16:0/4247686327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd1415dfd0 0x7fcd1406b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.351+0000 7fcd12d9d640 1 -- 172.21.15.16:0/4247686327 >> v2:172.21.15.156:3300/0 conn(0x7fcd1406bc40 msgr2=0x7fcd1406c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.351+0000 7fcd12d9d640 1 --2- 172.21.15.16:0/4247686327 >> v2:172.21.15.156:3300/0 conn(0x7fcd1406bc40 0x7fcd1406c0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.351+0000 7fcd12d9d640 1 -- 172.21.15.16:0/4247686327 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd140d2a90 con 0x7fcd140cefc0 2024-04-24T23:50:41.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.353+0000 7fcd12d9d640 1 --2- 172.21.15.16:0/4247686327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd140cefc0 0x7fcd140cf410 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fcd14067af0 tx=0x7fcd0c02a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef8637eaeb6d32c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.353+0000 7fcd1159a640 1 -- 172.21.15.16:0/4247686327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c036070 con 0x7fcd140cefc0 2024-04-24T23:50:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.353+0000 7fcd1159a640 1 -- 172.21.15.16:0/4247686327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd0c031600 con 0x7fcd140cefc0 2024-04-24T23:50:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.353+0000 7fcd1159a640 1 -- 172.21.15.16:0/4247686327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c0319d0 con 0x7fcd140cefc0 2024-04-24T23:50:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.353+0000 7fcd18e38640 1 -- 172.21.15.16:0/4247686327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd140cefc0 msgr2=0x7fcd140cf410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.353+0000 7fcd18e38640 1 --2- 172.21.15.16:0/4247686327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd140cefc0 0x7fcd140cf410 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fcd14067af0 tx=0x7fcd0c02a610 comp rx=0 tx=0).stop 2024-04-24T23:50:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.353+0000 7fcd18e38640 1 -- 172.21.15.16:0/4247686327 shutdown_connections 2024-04-24T23:50:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.353+0000 7fcd18e38640 1 --2- 172.21.15.16:0/4247686327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd140cefc0 0x7fcd140cf410 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.353+0000 7fcd18e38640 1 --2- 172.21.15.16:0/4247686327 >> v2:172.21.15.156:3300/0 conn(0x7fcd1406bc40 0x7fcd1406c0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.353+0000 7fcd18e38640 1 --2- 172.21.15.16:0/4247686327 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd1415dfd0 0x7fcd1406b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.353+0000 7fcd18e38640 1 -- 172.21.15.16:0/4247686327 >> 172.21.15.16:0/4247686327 conn(0x7fcd140d3dc0 msgr2=0x7fcd140d41d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.353+0000 7fcd18e38640 1 -- 172.21.15.16:0/4247686327 shutdown_connections 2024-04-24T23:50:41.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.353+0000 7fcd18e38640 1 -- 172.21.15.16:0/4247686327 wait complete. 2024-04-24T23:50:41.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.354+0000 7fcd18e38640 1 Processor -- start 2024-04-24T23:50:41.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.354+0000 7fcd18e38640 1 -- start start 2024-04-24T23:50:41.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.354+0000 7fcd18e38640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd1406bc40 0x7fcd14159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.356 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.354+0000 7fcd18e38640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd140cefc0 0x7fcd1415ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.354+0000 7fcd18e38640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd1415dfd0 0x7fcd140c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.354+0000 7fcd18e38640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fcd140cead0 con 0x7fcd1406bc40 2024-04-24T23:50:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.354+0000 7fcd18e38640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcd140d3050 con 0x7fcd1415dfd0 2024-04-24T23:50:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.354+0000 7fcd18e38640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fcd1406b830 con 0x7fcd140cefc0 2024-04-24T23:50:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd1259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd1406bc40 0x7fcd14159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd1259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd1406bc40 0x7fcd14159680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50956/0 (socket says 172.21.15.16:50956) 2024-04-24T23:50:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd1259c640 1 -- 172.21.15.16:0/3946485476 learned_addr learned my addr 172.21.15.16:0/3946485476 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd12d9d640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd1415dfd0 0x7fcd140c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd11d9b640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd140cefc0 0x7fcd1415ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd1259c640 1 -- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd140cefc0 msgr2=0x7fcd1415ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd1259c640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd140cefc0 0x7fcd1415ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd1259c640 1 -- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd1415dfd0 msgr2=0x7fcd140c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd1259c640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd1415dfd0 0x7fcd140c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd1259c640 1 -- 172.21.15.16:0/3946485476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd140c9f80 con 0x7fcd1406bc40 2024-04-24T23:50:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd12d9d640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd1415dfd0 0x7fcd140c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd11d9b640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd140cefc0 0x7fcd1415ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd1259c640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd1406bc40 0x7fcd14159680 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400e9e0 tx=0x7fcd0400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd02ffd640 1 -- 172.21.15.16:0/3946485476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd04017070 con 0x7fcd1406bc40 2024-04-24T23:50:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd02ffd640 1 -- 172.21.15.16:0/3946485476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd0400c9d0 con 0x7fcd1406bc40 2024-04-24T23:50:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd18e38640 1 -- 172.21.15.16:0/3946485476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd14159d80 con 0x7fcd1406bc40 2024-04-24T23:50:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd02ffd640 1 -- 172.21.15.16:0/3946485476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd0400cdc0 con 0x7fcd1406bc40 2024-04-24T23:50:41.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.355+0000 7fcd18e38640 1 -- 172.21.15.16:0/3946485476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd140cc130 con 0x7fcd1406bc40 2024-04-24T23:50:41.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.356+0000 7fcd18e38640 1 -- 172.21.15.16:0/3946485476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd14068f90 con 0x7fcd1406bc40 2024-04-24T23:50:41.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.357+0000 7fcd02ffd640 1 -- 172.21.15.16:0/3946485476 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fcd0401a070 con 0x7fcd1406bc40 2024-04-24T23:50:41.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.357+0000 7fcd02ffd640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fccf406be30 0x7fccf406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.357+0000 7fcd02ffd640 1 -- 172.21.15.16:0/3946485476 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7fcd0408af70 con 0x7fcd1406bc40 2024-04-24T23:50:41.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.358+0000 7fcd11d9b640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fccf406be30 0x7fccf406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.359 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.358+0000 7fcd11d9b640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fccf406be30 0x7fccf406e2f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcd1415d590 tx=0x7fcd08008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:41.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.362+0000 7fcd02ffd640 1 -- 172.21.15.16:0/3946485476 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fcd140c9f80 con 0x7fcd1406bc40 2024-04-24T23:50:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.542+0000 7fcd18e38640 1 -- 172.21.15.16:0/3946485476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcd140b6ff0 con 0x7fcd1406bc40 2024-04-24T23:50:41.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.542+0000 7fcd02ffd640 1 -- 172.21.15.16:0/3946485476 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v43) ==== 100+0+11 (secure 0 0 0) 0x7fcd040597b0 con 0x7fcd1406bc40 2024-04-24T23:50:41.543 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 64 2024-04-24T23:50:41.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.544+0000 7fcd18e38640 1 -- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fccf406be30 msgr2=0x7fccf406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.544+0000 7fcd18e38640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fccf406be30 0x7fccf406e2f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fcd1415d590 tx=0x7fcd08008040 comp rx=0 tx=0).stop 2024-04-24T23:50:41.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.544+0000 7fcd18e38640 1 -- 172.21.15.16:0/3946485476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd1406bc40 msgr2=0x7fcd14159680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.544+0000 7fcd18e38640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd1406bc40 0x7fcd14159680 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fcd0400e9e0 tx=0x7fcd0400eea0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.544+0000 7fcd18e38640 1 -- 172.21.15.16:0/3946485476 shutdown_connections 2024-04-24T23:50:41.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.544+0000 7fcd18e38640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fccf406be30 0x7fccf406e2f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.544+0000 7fcd18e38640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd1415dfd0 0x7fcd140c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.544+0000 7fcd18e38640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fcd140cefc0 0x7fcd1415ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.544+0000 7fcd18e38640 1 --2- 172.21.15.16:0/3946485476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd1406bc40 0x7fcd14159680 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.544+0000 7fcd18e38640 1 -- 172.21.15.16:0/3946485476 >> 172.21.15.16:0/3946485476 conn(0x7fcd140d3dc0 msgr2=0x7fcd140b9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:41.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.544+0000 7fcd18e38640 1 -- 172.21.15.16:0/3946485476 shutdown_connections 2024-04-24T23:50:41.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.544+0000 7fcd18e38640 1 -- 172.21.15.16:0/3946485476 wait complete. 2024-04-24T23:50:41.556 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:50:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.657+0000 7f7e32942640 1 Processor -- start 2024-04-24T23:50:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.657+0000 7f7e32942640 1 -- start start 2024-04-24T23:50:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e32942640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c06cb00 0x7f7e2c15c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e32942640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7e2c0ccc70 0x7f7e2c072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e32942640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e2c0cfd50 con 0x7f7e2c0727d0 2024-04-24T23:50:41.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e32942640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f7e2c0ce760 con 0x7f7e2c0ccc70 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e32942640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7e2c06cee0 con 0x7f7e2c06cb00 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e2bfff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c06cb00 0x7f7e2c15c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e2b7fe640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f7e2c0ccc70 0x7f7e2c072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e2bfff640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c06cb00 0x7f7e2c15c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:55864/0 (socket says 172.21.15.16:55864) 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e2bfff640 1 -- 172.21.15.16:0/2168055113 learned_addr learned my addr 172.21.15.16:0/2168055113 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e2b7fe640 1 -- 172.21.15.16:0/2168055113 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c06cb00 msgr2=0x7f7e2c15c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e2b7fe640 1 --2- 172.21.15.16:0/2168055113 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c06cb00 0x7f7e2c15c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e2b7fe640 1 -- 172.21.15.16:0/2168055113 >> v1:172.21.15.156:6789/0 conn(0x7f7e2c0727d0 legacy=0x7f7e2c15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e2b7fe640 1 -- 172.21.15.16:0/2168055113 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e2c15c8b0 con 0x7f7e2c0ccc70 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e2affd640 1 -- 172.21.15.16:0/2168055113 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 63469389 0 0) 0x7f7e2c0cfd50 con 0x7f7e2c0727d0 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e2bfff640 1 --2- 172.21.15.16:0/2168055113 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c06cb00 0x7f7e2c15c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e2b7fe640 1 --2- 172.21.15.16:0/2168055113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c0ccc70 0x7f7e2c072290 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7e200097a0 tx=0x7f7e2002a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d7fef07be4cf2b2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e2affd640 1 -- 172.21.15.16:0/2168055113 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e20037070 con 0x7f7e2c0ccc70 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.658+0000 7f7e2affd640 1 -- 172.21.15.16:0/2168055113 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e20032440 con 0x7f7e2c0ccc70 2024-04-24T23:50:41.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.659+0000 7f7e2affd640 1 -- 172.21.15.16:0/2168055113 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e20032810 con 0x7f7e2c0ccc70 2024-04-24T23:50:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.659+0000 7f7e32942640 1 -- 172.21.15.16:0/2168055113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c0ccc70 msgr2=0x7f7e2c072290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.659+0000 7f7e32942640 1 --2- 172.21.15.16:0/2168055113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c0ccc70 0x7f7e2c072290 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f7e200097a0 tx=0x7f7e2002a180 comp rx=0 tx=0).stop 2024-04-24T23:50:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.659+0000 7f7e32942640 1 -- 172.21.15.16:0/2168055113 shutdown_connections 2024-04-24T23:50:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.659+0000 7f7e32942640 1 --2- 172.21.15.16:0/2168055113 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c0ccc70 0x7f7e2c072290 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.663 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.659+0000 7f7e32942640 1 --2- 172.21.15.16:0/2168055113 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c06cb00 0x7f7e2c15c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.659+0000 7f7e32942640 1 -- 172.21.15.16:0/2168055113 >> 172.21.15.16:0/2168055113 conn(0x7f7e2c0d0260 msgr2=0x7f7e2c0d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.659+0000 7f7e32942640 1 -- 172.21.15.16:0/2168055113 shutdown_connections 2024-04-24T23:50:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.659+0000 7f7e32942640 1 -- 172.21.15.16:0/2168055113 wait complete. 2024-04-24T23:50:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.660+0000 7f7e32942640 1 Processor -- start 2024-04-24T23:50:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.661+0000 7f7e32942640 1 -- start start 2024-04-24T23:50:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.661+0000 7f7e32942640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2c06cb00 0x7f7e2c15af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.661+0000 7f7e32942640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c0727d0 0x7f7e2c0aea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.661+0000 7f7e32942640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c0ccc70 0x7f7e2c0aefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.661+0000 7f7e32942640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2c15d150 con 0x7f7e2c06cb00 2024-04-24T23:50:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.661+0000 7f7e32942640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2c15ce70 con 0x7f7e2c0727d0 2024-04-24T23:50:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.661+0000 7f7e32942640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f7e2c06b0e0 con 0x7f7e2c0ccc70 2024-04-24T23:50:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.661+0000 7f7e2bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2c06cb00 0x7f7e2c15af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.661+0000 7f7e2bfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2c06cb00 0x7f7e2c15af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50968/0 (socket says 172.21.15.16:50968) 2024-04-24T23:50:41.664 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.661+0000 7f7e2bfff640 1 -- 172.21.15.16:0/1896862930 learned_addr learned my addr 172.21.15.16:0/1896862930 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.662+0000 7f7e2b7fe640 1 --2- 172.21.15.16:0/1896862930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c0727d0 0x7f7e2c0aea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.662+0000 7f7e30edf640 1 --2- 172.21.15.16:0/1896862930 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c0ccc70 0x7f7e2c0aefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.662+0000 7f7e2bfff640 1 -- 172.21.15.16:0/1896862930 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c0ccc70 msgr2=0x7f7e2c0aefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.662+0000 7f7e2bfff640 1 --2- 172.21.15.16:0/1896862930 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c0ccc70 0x7f7e2c0aefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.662+0000 7f7e2bfff640 1 -- 172.21.15.16:0/1896862930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c0727d0 msgr2=0x7f7e2c0aea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.662+0000 7f7e2bfff640 1 --2- 172.21.15.16:0/1896862930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c0727d0 0x7f7e2c0aea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.662+0000 7f7e2bfff640 1 -- 172.21.15.16:0/1896862930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e2c1f7370 con 0x7f7e2c06cb00 2024-04-24T23:50:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.662+0000 7f7e2bfff640 1 --2- 172.21.15.16:0/1896862930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2c06cb00 0x7f7e2c15af40 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00e9b0 tx=0x7f7e1c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.662+0000 7f7e28ff9640 1 -- 172.21.15.16:0/1896862930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c017070 con 0x7f7e2c06cb00 2024-04-24T23:50:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.662+0000 7f7e28ff9640 1 -- 172.21.15.16:0/1896862930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e1c00c9a0 con 0x7f7e2c06cb00 2024-04-24T23:50:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.662+0000 7f7e32942640 1 -- 172.21.15.16:0/1896862930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e2c1f7610 con 0x7f7e2c06cb00 2024-04-24T23:50:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.662+0000 7f7e28ff9640 1 -- 172.21.15.16:0/1896862930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1c00cd50 con 0x7f7e2c06cb00 2024-04-24T23:50:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.662+0000 7f7e32942640 1 -- 172.21.15.16:0/1896862930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e2c1f7bd0 con 0x7f7e2c06cb00 2024-04-24T23:50:41.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.663+0000 7f7e32942640 1 -- 172.21.15.16:0/1896862930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e2c0cd0f0 con 0x7f7e2c06cb00 2024-04-24T23:50:41.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.664+0000 7f7e28ff9640 1 -- 172.21.15.16:0/1896862930 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f7e1c01a070 con 0x7f7e2c06cb00 2024-04-24T23:50:41.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.664+0000 7f7e28ff9640 1 --2- 172.21.15.16:0/1896862930 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7e1806be30 0x7f7e1806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:41.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.664+0000 7f7e28ff9640 1 -- 172.21.15.16:0/1896862930 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7f7e1c08b2b0 con 0x7f7e2c06cb00 2024-04-24T23:50:41.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.664+0000 7f7e2b7fe640 1 --2- 172.21.15.16:0/1896862930 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7e1806be30 0x7f7e1806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:41.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.665+0000 7f7e2b7fe640 1 --2- 172.21.15.16:0/1896862930 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7e1806be30 0x7f7e1806e2f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7e20034000 tx=0x7f7e2002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:41.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.668+0000 7f7e28ff9640 1 -- 172.21.15.16:0/1896862930 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f7e2c0cd0f0 con 0x7f7e2c06cb00 2024-04-24T23:50:41.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.892+0000 7f7e32942640 1 -- 172.21.15.16:0/1896862930 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7e2c04f5e0 con 0x7f7e2c06cb00 2024-04-24T23:50:41.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.892+0000 7f7e28ff9640 1 -- 172.21.15.16:0/1896862930 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5184 (secure 0 0 0) 0x7f7e1c059a70 con 0x7f7e2c06cb00 2024-04-24T23:50:41.892 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:41.892 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5299,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/448535992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":448535992},{"type":"v1","addr":"172.21.15.16:6835","nonce":448535992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5305,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1890687780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1890687780},{"type":"v1","addr":"172.21.15.156:6836","nonce":1890687780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5308,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2538147395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":2538147395},{"type":"v1","addr":"172.21.15.156:6837","nonce":2538147395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:50:31.671778+0000","modified":"2024-04-24T23:50:35.722577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5736},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5736":{"gid":5736,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6834/3643102469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3643102469},{"type":"v1","addr":"172.21.15.16:6834","nonce":3643102469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":5736,"qdb_cluster":[5736]},"id":2}]} 2024-04-24T23:50:41.893 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 23 2024-04-24T23:50:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.895+0000 7f7e32942640 1 -- 172.21.15.16:0/1896862930 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7e1806be30 msgr2=0x7f7e1806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.895+0000 7f7e32942640 1 --2- 172.21.15.16:0/1896862930 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7e1806be30 0x7f7e1806e2f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7e20034000 tx=0x7f7e2002e000 comp rx=0 tx=0).stop 2024-04-24T23:50:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.895+0000 7f7e32942640 1 -- 172.21.15.16:0/1896862930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2c06cb00 msgr2=0x7f7e2c15af40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.895+0000 7f7e32942640 1 --2- 172.21.15.16:0/1896862930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2c06cb00 0x7f7e2c15af40 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f7e1c00e9b0 tx=0x7f7e1c00ee70 comp rx=0 tx=0).stop 2024-04-24T23:50:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.895+0000 7f7e32942640 1 -- 172.21.15.16:0/1896862930 shutdown_connections 2024-04-24T23:50:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.895+0000 7f7e32942640 1 --2- 172.21.15.16:0/1896862930 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f7e1806be30 0x7f7e1806e2f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.895+0000 7f7e32942640 1 --2- 172.21.15.16:0/1896862930 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f7e2c0ccc70 0x7f7e2c0aefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.895+0000 7f7e32942640 1 --2- 172.21.15.16:0/1896862930 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7e2c0727d0 0x7f7e2c0aea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.895+0000 7f7e32942640 1 --2- 172.21.15.16:0/1896862930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2c06cb00 0x7f7e2c15af40 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.895+0000 7f7e32942640 1 -- 172.21.15.16:0/1896862930 >> 172.21.15.16:0/1896862930 conn(0x7f7e2c0d0260 msgr2=0x7f7e2c0b0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.895+0000 7f7e32942640 1 -- 172.21.15.16:0/1896862930 shutdown_connections 2024-04-24T23:50:41.895 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:41.895+0000 7f7e32942640 1 -- 172.21.15.16:0/1896862930 wait complete. 2024-04-24T23:50:41.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:50:42.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.007+0000 7fc30c48c640 1 Processor -- start 2024-04-24T23:50:42.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.007+0000 7fc30c48c640 1 -- start start 2024-04-24T23:50:42.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.008+0000 7fc30c48c640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fc30415dfa0 0x7fc3040d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.008+0000 7fc30c48c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3040ce5f0 0x7fc3040cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.008+0000 7fc30c48c640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3040d05a0 con 0x7fc3040b5f00 2024-04-24T23:50:42.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.008+0000 7fc30c48c640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fc3040b7360 con 0x7fc30415dfa0 2024-04-24T23:50:42.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.008+0000 7fc30c48c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fc3040cc640 con 0x7fc3040ce5f0 2024-04-24T23:50:42.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.008+0000 7fc30aa29640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc3040b5f00 0x7fc3040b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32918/0 (socket says 172.21.15.16:32918) 2024-04-24T23:50:42.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.008+0000 7fc30aa29640 1 -- 172.21.15.16:0/184709482 learned_addr learned my addr 172.21.15.16:0/184709482 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:42.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.008+0000 7fc30a228640 1 --2- 172.21.15.16:0/184709482 >> v2:172.21.15.156:3300/0 conn(0x7fc30415dfa0 0x7fc3040d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.008+0000 7fc309a27640 1 --2- 172.21.15.16:0/184709482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3040ce5f0 0x7fc3040cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.008+0000 7fc309226640 1 -- 172.21.15.16:0/184709482 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3273521914 0 0) 0x7fc3040d05a0 con 0x7fc3040b5f00 2024-04-24T23:50:42.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.008+0000 7fc309226640 1 -- 172.21.15.16:0/184709482 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc2ec003520 con 0x7fc3040b5f00 2024-04-24T23:50:42.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.009+0000 7fc30a228640 1 -- 172.21.15.16:0/184709482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3040ce5f0 msgr2=0x7fc3040cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:42.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.009+0000 7fc30a228640 1 --2- 172.21.15.16:0/184709482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3040ce5f0 0x7fc3040cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.009+0000 7fc30a228640 1 -- 172.21.15.16:0/184709482 >> v1:172.21.15.16:6789/0 conn(0x7fc3040b5f00 legacy=0x7fc3040b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.009 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.009+0000 7fc30a228640 1 -- 172.21.15.16:0/184709482 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3040d0880 con 0x7fc30415dfa0 2024-04-24T23:50:42.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.009+0000 7fc30a228640 1 --2- 172.21.15.16:0/184709482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc30415dfa0 0x7fc3040d0fe0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc0097a0 tx=0x7fc2fc02a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca07726435b39ca0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:42.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.010+0000 7fc309226640 1 -- 172.21.15.16:0/184709482 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc037070 con 0x7fc30415dfa0 2024-04-24T23:50:42.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.010+0000 7fc309226640 1 -- 172.21.15.16:0/184709482 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2fc032440 con 0x7fc30415dfa0 2024-04-24T23:50:42.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.010+0000 7fc309226640 1 -- 172.21.15.16:0/184709482 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2fc032810 con 0x7fc30415dfa0 2024-04-24T23:50:42.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.010+0000 7fc30c48c640 1 -- 172.21.15.16:0/184709482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc30415dfa0 msgr2=0x7fc3040d0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.010+0000 7fc30c48c640 1 --2- 172.21.15.16:0/184709482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc30415dfa0 0x7fc3040d0fe0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc0097a0 tx=0x7fc2fc02a180 comp rx=0 tx=0).stop 2024-04-24T23:50:42.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.010+0000 7fc30c48c640 1 -- 172.21.15.16:0/184709482 shutdown_connections 2024-04-24T23:50:42.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.010+0000 7fc30c48c640 1 --2- 172.21.15.16:0/184709482 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3040ce5f0 0x7fc3040cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.010+0000 7fc30c48c640 1 --2- 172.21.15.16:0/184709482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc30415dfa0 0x7fc3040d0fe0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.010+0000 7fc30c48c640 1 -- 172.21.15.16:0/184709482 >> 172.21.15.16:0/184709482 conn(0x7fc30406b4d0 msgr2=0x7fc3040b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:42.011 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.010+0000 7fc30c48c640 1 -- 172.21.15.16:0/184709482 shutdown_connections 2024-04-24T23:50:42.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.010+0000 7fc30c48c640 1 -- 172.21.15.16:0/184709482 wait complete. 2024-04-24T23:50:42.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.011+0000 7fc30c48c640 1 Processor -- start 2024-04-24T23:50:42.012 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.011+0000 7fc30c48c640 1 -- start start 2024-04-24T23:50:42.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc30c48c640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3040b5f00 0x7fc30414a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.014 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc30c48c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3040ce5f0 0x7fc3040c65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc30c48c640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc30415dfa0 0x7fc30414dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc30c48c640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fc30415e380 con 0x7fc3040ce5f0 2024-04-24T23:50:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc30c48c640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc3040cc2d0 con 0x7fc30415dfa0 2024-04-24T23:50:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc30c48c640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fc3040d1520 con 0x7fc3040b5f00 2024-04-24T23:50:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc309a27640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3040ce5f0 0x7fc3040c65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc309a27640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3040ce5f0 0x7fc3040c65b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50980/0 (socket says 172.21.15.16:50980) 2024-04-24T23:50:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc309a27640 1 -- 172.21.15.16:0/842397126 learned_addr learned my addr 172.21.15.16:0/842397126 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc30a228640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3040b5f00 0x7fc30414a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc30aa29640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc30415dfa0 0x7fc30414dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc309a27640 1 -- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3040b5f00 msgr2=0x7fc30414a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc309a27640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3040b5f00 0x7fc30414a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc309a27640 1 -- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc30415dfa0 msgr2=0x7fc30414dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.012+0000 7fc309a27640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc30415dfa0 0x7fc30414dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.013+0000 7fc309a27640 1 -- 172.21.15.16:0/842397126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc30406cf60 con 0x7fc3040ce5f0 2024-04-24T23:50:42.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.013+0000 7fc30a228640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3040b5f00 0x7fc30414a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:42.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.013+0000 7fc309a27640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3040ce5f0 0x7fc3040c65b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800e9e0 tx=0x7fc2f800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:42.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.013+0000 7fc302ffd640 1 -- 172.21.15.16:0/842397126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f8017070 con 0x7fc3040ce5f0 2024-04-24T23:50:42.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.013+0000 7fc302ffd640 1 -- 172.21.15.16:0/842397126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f800c9d0 con 0x7fc3040ce5f0 2024-04-24T23:50:42.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.013+0000 7fc302ffd640 1 -- 172.21.15.16:0/842397126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f800cdc0 con 0x7fc3040ce5f0 2024-04-24T23:50:42.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.013+0000 7fc30c48c640 1 -- 172.21.15.16:0/842397126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc30406d240 con 0x7fc3040ce5f0 2024-04-24T23:50:42.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.013+0000 7fc30c48c640 1 -- 172.21.15.16:0/842397126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc30406d800 con 0x7fc3040ce5f0 2024-04-24T23:50:42.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.014+0000 7fc30c48c640 1 -- 172.21.15.16:0/842397126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc30406bea0 con 0x7fc3040ce5f0 2024-04-24T23:50:42.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.014+0000 7fc302ffd640 1 -- 172.21.15.16:0/842397126 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc2f801a070 con 0x7fc3040ce5f0 2024-04-24T23:50:42.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.015+0000 7fc302ffd640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc2f006be30 0x7fc2f006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.015+0000 7fc302ffd640 1 -- 172.21.15.16:0/842397126 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7fc2f808b420 con 0x7fc3040ce5f0 2024-04-24T23:50:42.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.015+0000 7fc30a228640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc2f006be30 0x7fc2f006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.016+0000 7fc30a228640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc2f006be30 0x7fc2f006e2f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc034000 tx=0x7fc2fc02e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.019+0000 7fc302ffd640 1 -- 172.21.15.16:0/842397126 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fc2f8059500 con 0x7fc3040ce5f0 2024-04-24T23:50:42.203 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.203+0000 7fc30c48c640 1 -- 172.21.15.16:0/842397126 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc30415d730 con 0x7fc3040ce5f0 2024-04-24T23:50:42.204 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.204+0000 7fc302ffd640 1 -- 172.21.15.16:0/842397126 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+17050 (secure 0 0 0) 0x7fc2f805d800 con 0x7fc3040ce5f0 2024-04-24T23:50:42.204 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:42.205 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":43,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:50:37.287284+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:50:17.026573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-24T23:50:17.415963+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.36","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6834/561619268":"2024-04-25T23:50:30.259891+0000","172.21.15.16:6832/561619268":"2024-04-25T23:50:30.259891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:50:42.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.206+0000 7fc30c48c640 1 -- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc2f006be30 msgr2=0x7fc2f006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.206+0000 7fc30c48c640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc2f006be30 0x7fc2f006e2f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc034000 tx=0x7fc2fc02e000 comp rx=0 tx=0).stop 2024-04-24T23:50:42.206 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.206+0000 7fc30c48c640 1 -- 172.21.15.16:0/842397126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3040ce5f0 msgr2=0x7fc3040c65b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.206+0000 7fc30c48c640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3040ce5f0 0x7fc3040c65b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc2f800e9e0 tx=0x7fc2f800eea0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.206+0000 7fc30c48c640 1 -- 172.21.15.16:0/842397126 shutdown_connections 2024-04-24T23:50:42.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.206+0000 7fc30c48c640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fc2f006be30 0x7fc2f006e2f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.206+0000 7fc30c48c640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc30415dfa0 0x7fc30414dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.206+0000 7fc30c48c640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3040ce5f0 0x7fc3040c65b0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.206+0000 7fc30c48c640 1 --2- 172.21.15.16:0/842397126 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fc3040b5f00 0x7fc30414a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.206+0000 7fc30c48c640 1 -- 172.21.15.16:0/842397126 >> 172.21.15.16:0/842397126 conn(0x7fc30406b4d0 msgr2=0x7fc3040c47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:42.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.207+0000 7fc30c48c640 1 -- 172.21.15.16:0/842397126 shutdown_connections 2024-04-24T23:50:42.207 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.207+0000 7fc30c48c640 1 -- 172.21.15.16:0/842397126 wait complete. 2024-04-24T23:50:42.219 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T23:50:42.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:50:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.322+0000 7f1e22928640 1 Processor -- start 2024-04-24T23:50:42.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.322+0000 7f1e22928640 1 -- start start 2024-04-24T23:50:42.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.322+0000 7f1e22928640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f1e1c06eaf0 0x7f1e1c06c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.322+0000 7f1e22928640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e1c15f890 con 0x7f1e1c13eba0 2024-04-24T23:50:42.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.322+0000 7f1e22928640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e1c15fb50 con 0x7f1e1c072110 2024-04-24T23:50:42.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.322+0000 7f1e22928640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f1e1c160480 con 0x7f1e1c06eaf0 2024-04-24T23:50:42.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1e1c13eba0 0x7f1e1c071a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32932/0 (socket says 172.21.15.16:32932) 2024-04-24T23:50:42.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1bfff640 1 -- 172.21.15.16:0/3788274050 learned_addr learned my addr 172.21.15.16:0/3788274050 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:42.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e20ec5640 1 --2- 172.21.15.16:0/3788274050 >> v2:172.21.15.156:3300/0 conn(0x7f1e1c06eaf0 0x7f1e1c06c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 -- 172.21.15.16:0/3788274050 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3241462760 0 0) 0x7f1e1c15f890 con 0x7f1e1c13eba0 2024-04-24T23:50:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 -- 172.21.15.16:0/3788274050 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e04003520 con 0x7f1e1c13eba0 2024-04-24T23:50:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 -- 172.21.15.16:0/3788274050 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2173708207 0 0) 0x7f1e04003520 con 0x7f1e1c13eba0 2024-04-24T23:50:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 -- 172.21.15.16:0/3788274050 >> v2:172.21.15.156:3300/0 conn(0x7f1e1c06eaf0 msgr2=0x7f1e1c06c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 --2- 172.21.15.16:0/3788274050 >> v2:172.21.15.156:3300/0 conn(0x7f1e1c06eaf0 0x7f1e1c06c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 -- 172.21.15.16:0/3788274050 >> v1:172.21.15.156:6789/0 conn(0x7f1e1c072110 legacy=0x7f1e1c06ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 -- 172.21.15.16:0/3788274050 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e1c15fe30 con 0x7f1e1c13eba0 2024-04-24T23:50:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 -- 172.21.15.16:0/3788274050 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f1e0c001670 con 0x7f1e1c13eba0 2024-04-24T23:50:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 -- 172.21.15.16:0/3788274050 >> v1:172.21.15.16:6789/0 conn(0x7f1e1c13eba0 legacy=0x7f1e1c071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 --2- 172.21.15.16:0/3788274050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e040039b0 0x7f1e04003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 --2- 172.21.15.16:0/3788274050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e04004800 0x7f1e04004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 --2- 172.21.15.16:0/3788274050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e040051c0 0x7f1e0400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 -- 172.21.15.16:0/3788274050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c160480 con 0x7f1e040051c0 2024-04-24T23:50:42.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 -- 172.21.15.16:0/3788274050 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1e0400c090 con 0x7f1e04004800 2024-04-24T23:50:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1affd640 1 -- 172.21.15.16:0/3788274050 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1e0400c290 con 0x7f1e040039b0 2024-04-24T23:50:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e20ec5640 1 --2- 172.21.15.16:0/3788274050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e040051c0 0x7f1e0400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1bfff640 1 --2- 172.21.15.16:0/3788274050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e040039b0 0x7f1e04003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.323+0000 7f1e1b7fe640 1 --2- 172.21.15.16:0/3788274050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e04004800 0x7f1e04004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.324+0000 7f1e20ec5640 1 -- 172.21.15.16:0/3788274050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e040039b0 msgr2=0x7f1e04003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.324+0000 7f1e20ec5640 1 --2- 172.21.15.16:0/3788274050 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e040039b0 0x7f1e04003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.324+0000 7f1e20ec5640 1 -- 172.21.15.16:0/3788274050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e04004800 msgr2=0x7f1e04004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.324+0000 7f1e20ec5640 1 --2- 172.21.15.16:0/3788274050 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e04004800 0x7f1e04004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.324+0000 7f1e20ec5640 1 -- 172.21.15.16:0/3788274050 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e04003520 con 0x7f1e040051c0 2024-04-24T23:50:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.324+0000 7f1e20ec5640 1 -- 172.21.15.16:0/3788274050 reap_dead start 2024-04-24T23:50:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.324+0000 7f1e20ec5640 1 --2- 172.21.15.16:0/3788274050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e040051c0 0x7f1e0400b9c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1e14009fc0 tx=0x7f1e04005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a57f9dde2720aa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.325+0000 7f1e1affd640 1 -- 172.21.15.16:0/3788274050 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e14009c10 con 0x7f1e040051c0 2024-04-24T23:50:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.325+0000 7f1e1affd640 1 -- 172.21.15.16:0/3788274050 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e14003990 con 0x7f1e040051c0 2024-04-24T23:50:42.326 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.325+0000 7f1e1affd640 1 -- 172.21.15.16:0/3788274050 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e1402d070 con 0x7f1e040051c0 2024-04-24T23:50:42.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.325+0000 7f1e22928640 1 -- 172.21.15.16:0/3788274050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e040051c0 msgr2=0x7f1e0400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.325+0000 7f1e22928640 1 --2- 172.21.15.16:0/3788274050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e040051c0 0x7f1e0400b9c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1e14009fc0 tx=0x7f1e04005830 comp rx=0 tx=0).stop 2024-04-24T23:50:42.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.325+0000 7f1e22928640 1 -- 172.21.15.16:0/3788274050 shutdown_connections 2024-04-24T23:50:42.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.325+0000 7f1e22928640 1 --2- 172.21.15.16:0/3788274050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e040051c0 0x7f1e0400b9c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.325+0000 7f1e22928640 1 -- 172.21.15.16:0/3788274050 >> 172.21.15.16:0/3788274050 conn(0x7f1e1c0d0ac0 msgr2=0x7f1e1c0d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:42.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.325+0000 7f1e22928640 1 -- 172.21.15.16:0/3788274050 shutdown_connections 2024-04-24T23:50:42.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.325+0000 7f1e22928640 1 -- 172.21.15.16:0/3788274050 wait complete. 2024-04-24T23:50:42.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.326+0000 7f1e22928640 1 Processor -- start 2024-04-24T23:50:42.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.326+0000 7f1e22928640 1 -- start start 2024-04-24T23:50:42.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.327+0000 7f1e22928640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e040051c0 0x7f1e1c0b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.327+0000 7f1e22928640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e1c155ca0 0x7f1e1c0b5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.327+0000 7f1e22928640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e1c158ce0 0x7f1e1c159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.327+0000 7f1e22928640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c071a00 con 0x7f1e1c155ca0 2024-04-24T23:50:42.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.327+0000 7f1e22928640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1e1c13ef80 con 0x7f1e040051c0 2024-04-24T23:50:42.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.327+0000 7f1e22928640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1e1c071b80 con 0x7f1e1c158ce0 2024-04-24T23:50:42.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.327+0000 7f1e1b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e1c155ca0 0x7f1e1c0b5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.327+0000 7f1e1b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e1c155ca0 0x7f1e1c0b5ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51002/0 (socket says 172.21.15.16:51002) 2024-04-24T23:50:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.327+0000 7f1e1b7fe640 1 -- 172.21.15.16:0/1673434323 learned_addr learned my addr 172.21.15.16:0/1673434323 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.327+0000 7f1e1bfff640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e040051c0 0x7f1e1c0b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.327+0000 7f1e20ec5640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e1c158ce0 0x7f1e1c159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.328+0000 7f1e1b7fe640 1 -- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e1c158ce0 msgr2=0x7f1e1c159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.328+0000 7f1e1b7fe640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e1c158ce0 0x7f1e1c159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.328+0000 7f1e1b7fe640 1 -- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e040051c0 msgr2=0x7f1e1c0b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.328+0000 7f1e1b7fe640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e040051c0 0x7f1e1c0b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.328+0000 7f1e1b7fe640 1 -- 172.21.15.16:0/1673434323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e1c140c70 con 0x7f1e1c155ca0 2024-04-24T23:50:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.328+0000 7f1e20ec5640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e1c158ce0 0x7f1e1c159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.328+0000 7f1e1b7fe640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e1c155ca0 0x7f1e1c0b5ec0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1e1000f650 tx=0x7f1e1000fb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.328+0000 7f1e18ff9640 1 -- 172.21.15.16:0/1673434323 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e100050d0 con 0x7f1e1c155ca0 2024-04-24T23:50:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.328+0000 7f1e18ff9640 1 -- 172.21.15.16:0/1673434323 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e10018070 con 0x7f1e1c155ca0 2024-04-24T23:50:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.328+0000 7f1e18ff9640 1 -- 172.21.15.16:0/1673434323 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e10005490 con 0x7f1e1c155ca0 2024-04-24T23:50:42.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.328+0000 7f1e22928640 1 -- 172.21.15.16:0/1673434323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e1c159aa0 con 0x7f1e1c155ca0 2024-04-24T23:50:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.328+0000 7f1e22928640 1 -- 172.21.15.16:0/1673434323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e1c15a060 con 0x7f1e1c155ca0 2024-04-24T23:50:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.329+0000 7f1e22928640 1 -- 172.21.15.16:0/1673434323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e1c06f8f0 con 0x7f1e1c155ca0 2024-04-24T23:50:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.329+0000 7f1e18ff9640 1 -- 172.21.15.16:0/1673434323 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1e1000d6d0 con 0x7f1e1c155ca0 2024-04-24T23:50:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.330+0000 7f1e18ff9640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1e0806be60 0x7f1e0806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.330+0000 7f1e18ff9640 1 -- 172.21.15.16:0/1673434323 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7f1e1008ac10 con 0x7f1e1c155ca0 2024-04-24T23:50:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.330+0000 7f1e1bfff640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1e0806be60 0x7f1e0806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.331+0000 7f1e1bfff640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1e0806be60 0x7f1e0806e320 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c001050 tx=0x7f1e0c0078f0 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.334+0000 7f1e18ff9640 1 -- 172.21.15.16:0/1673434323 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f1e10059450 con 0x7f1e1c155ca0 2024-04-24T23:50:42.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.555+0000 7f1e22928640 1 -- 172.21.15.16:0/1673434323 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e1c04f5e0 con 0x7f1e1c155ca0 2024-04-24T23:50:42.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.556+0000 7f1e18ff9640 1 -- 172.21.15.16:0/1673434323 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5184 (secure 0 0 0) 0x7f1e10058d50 con 0x7f1e1c155ca0 2024-04-24T23:50:42.555 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:42.555 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5299,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/448535992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":448535992},{"type":"v1","addr":"172.21.15.16:6835","nonce":448535992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5305,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1890687780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1890687780},{"type":"v1","addr":"172.21.15.156:6836","nonce":1890687780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5308,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2538147395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":2538147395},{"type":"v1","addr":"172.21.15.156:6837","nonce":2538147395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:50:31.671778+0000","modified":"2024-04-24T23:50:35.722577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5736},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5736":{"gid":5736,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6834/3643102469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3643102469},{"type":"v1","addr":"172.21.15.16:6834","nonce":3643102469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":5736,"qdb_cluster":[5736]},"id":2}]} 2024-04-24T23:50:42.556 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 23 2024-04-24T23:50:42.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.557+0000 7f1e22928640 1 -- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1e0806be60 msgr2=0x7f1e0806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.557+0000 7f1e22928640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1e0806be60 0x7f1e0806e320 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c001050 tx=0x7f1e0c0078f0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.557+0000 7f1e22928640 1 -- 172.21.15.16:0/1673434323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e1c155ca0 msgr2=0x7f1e1c0b5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.557 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.557+0000 7f1e22928640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e1c155ca0 0x7f1e1c0b5ec0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1e1000f650 tx=0x7f1e1000fb10 comp rx=0 tx=0).stop 2024-04-24T23:50:42.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.558+0000 7f1e22928640 1 -- 172.21.15.16:0/1673434323 shutdown_connections 2024-04-24T23:50:42.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.558+0000 7f1e22928640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f1e0806be60 0x7f1e0806e320 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.558+0000 7f1e22928640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f1e1c158ce0 0x7f1e1c159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.558+0000 7f1e22928640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e1c155ca0 0x7f1e1c0b5ec0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.558+0000 7f1e22928640 1 --2- 172.21.15.16:0/1673434323 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e040051c0 0x7f1e1c0b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.558+0000 7f1e22928640 1 -- 172.21.15.16:0/1673434323 >> 172.21.15.16:0/1673434323 conn(0x7f1e1c0d0ac0 msgr2=0x7f1e1c13c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:42.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.558+0000 7f1e22928640 1 -- 172.21.15.16:0/1673434323 shutdown_connections 2024-04-24T23:50:42.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.558+0000 7f1e22928640 1 -- 172.21.15.16:0/1673434323 wait complete. 2024-04-24T23:50:42.569 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:50:42.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.669+0000 7f959a06b640 1 Processor -- start 2024-04-24T23:50:42.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.669+0000 7f959a06b640 1 -- start start 2024-04-24T23:50:42.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.670+0000 7f959a06b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95940ced90 0x7f9594166d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.670+0000 7f959a06b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959406ea50 con 0x7f95941457b0 2024-04-24T23:50:42.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.670+0000 7f959a06b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f959415fe90 con 0x7f9594144b20 2024-04-24T23:50:42.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.670+0000 7f959a06b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9594160170 con 0x7f95940ced90 2024-04-24T23:50:42.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.670+0000 7f9593fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f95940ced90 0x7f9594166d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.670+0000 7f9592ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f95941457b0 0x7f95940ce680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32946/0 (socket says 172.21.15.16:32946) 2024-04-24T23:50:42.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.670+0000 7f9592ffd640 1 -- 172.21.15.16:0/2820802598 learned_addr learned my addr 172.21.15.16:0/2820802598 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:42.670 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.670+0000 7f9593fff640 1 -- 172.21.15.16:0/2820802598 >> v1:172.21.15.156:6789/0 conn(0x7f9594144b20 legacy=0x7f9594144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.670+0000 7f9593fff640 1 -- 172.21.15.16:0/2820802598 >> v1:172.21.15.16:6789/0 conn(0x7f95941457b0 legacy=0x7f95940ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.670+0000 7f9593fff640 1 -- 172.21.15.16:0/2820802598 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9594160450 con 0x7f95940ced90 2024-04-24T23:50:42.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.670+0000 7f95927fc640 1 -- 172.21.15.16:0/2820802598 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 529855759 0 0) 0x7f959406ea50 con 0x7f95941457b0 2024-04-24T23:50:42.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.671+0000 7f9593fff640 1 --2- 172.21.15.16:0/2820802598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95940ced90 0x7f9594166d00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f958c009d90 tx=0x7f958c02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d6acabf46a010b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:42.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.671+0000 7f95927fc640 1 -- 172.21.15.16:0/2820802598 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c007740 con 0x7f95940ced90 2024-04-24T23:50:42.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.671+0000 7f95927fc640 1 -- 172.21.15.16:0/2820802598 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f958c0078e0 con 0x7f95940ced90 2024-04-24T23:50:42.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.671+0000 7f95927fc640 1 -- 172.21.15.16:0/2820802598 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f958c007cb0 con 0x7f95940ced90 2024-04-24T23:50:42.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.672+0000 7f959a06b640 1 -- 172.21.15.16:0/2820802598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95940ced90 msgr2=0x7f9594166d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.672+0000 7f959a06b640 1 --2- 172.21.15.16:0/2820802598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95940ced90 0x7f9594166d00 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f958c009d90 tx=0x7f958c02a660 comp rx=0 tx=0).stop 2024-04-24T23:50:42.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.672+0000 7f959a06b640 1 -- 172.21.15.16:0/2820802598 shutdown_connections 2024-04-24T23:50:42.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.672+0000 7f959a06b640 1 --2- 172.21.15.16:0/2820802598 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f95940ced90 0x7f9594166d00 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.672+0000 7f959a06b640 1 -- 172.21.15.16:0/2820802598 >> 172.21.15.16:0/2820802598 conn(0x7f95940b2000 msgr2=0x7f95940cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.672+0000 7f959a06b640 1 -- 172.21.15.16:0/2820802598 shutdown_connections 2024-04-24T23:50:42.673 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.672+0000 7f959a06b640 1 -- 172.21.15.16:0/2820802598 wait complete. 2024-04-24T23:50:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.673+0000 7f959a06b640 1 Processor -- start 2024-04-24T23:50:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.673+0000 7f959a06b640 1 -- start start 2024-04-24T23:50:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f959a06b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95940ced90 0x7f9594133e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f959a06b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9594144b20 0x7f95941300b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f959a06b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95941457b0 0x7f95940cb250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f959a06b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f95940b3390 con 0x7f9594144b20 2024-04-24T23:50:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f959a06b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f959406d4b0 con 0x7f95941457b0 2024-04-24T23:50:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f959a06b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f959415f990 con 0x7f95940ced90 2024-04-24T23:50:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f9592ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9594144b20 0x7f95941300b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f9592ffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9594144b20 0x7f95941300b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51026/0 (socket says 172.21.15.16:51026) 2024-04-24T23:50:42.675 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f9592ffd640 1 -- 172.21.15.16:0/3004686432 learned_addr learned my addr 172.21.15.16:0/3004686432 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f9593fff640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95941457b0 0x7f95940cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f95937fe640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95940ced90 0x7f9594133e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f9592ffd640 1 -- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95940ced90 msgr2=0x7f9594133e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f9592ffd640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95940ced90 0x7f9594133e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f9592ffd640 1 -- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95941457b0 msgr2=0x7f95940cb250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f9592ffd640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95941457b0 0x7f95940cb250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f9592ffd640 1 -- 172.21.15.16:0/3004686432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9594158d00 con 0x7f9594144b20 2024-04-24T23:50:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f9593fff640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95941457b0 0x7f95940cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.674+0000 7f9592ffd640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9594144b20 0x7f95941300b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f958800c3c0 tx=0x7f958800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.675+0000 7f957bfff640 1 -- 172.21.15.16:0/3004686432 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95880074e0 con 0x7f9594144b20 2024-04-24T23:50:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.675+0000 7f957bfff640 1 -- 172.21.15.16:0/3004686432 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95880047e0 con 0x7f9594144b20 2024-04-24T23:50:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.675+0000 7f959a06b640 1 -- 172.21.15.16:0/3004686432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9594158fe0 con 0x7f9594144b20 2024-04-24T23:50:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.675+0000 7f957bfff640 1 -- 172.21.15.16:0/3004686432 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9588004bd0 con 0x7f9594144b20 2024-04-24T23:50:42.676 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.675+0000 7f959a06b640 1 -- 172.21.15.16:0/3004686432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95941595a0 con 0x7f9594144b20 2024-04-24T23:50:42.677 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.676+0000 7f959a06b640 1 -- 172.21.15.16:0/3004686432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95940731f0 con 0x7f9594144b20 2024-04-24T23:50:42.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.676+0000 7f957bfff640 1 -- 172.21.15.16:0/3004686432 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9588011070 con 0x7f9594144b20 2024-04-24T23:50:42.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.677+0000 7f957bfff640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f958006be30 0x7f958006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:42.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.677+0000 7f957bfff640 1 -- 172.21.15.16:0/3004686432 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7f958808abe0 con 0x7f9594144b20 2024-04-24T23:50:42.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.677+0000 7f95937fe640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f958006be30 0x7f958006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:42.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.677+0000 7f95937fe640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f958006be30 0x7f958006e2f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9584000f60 tx=0x7f9584009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:42.681 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.681+0000 7f957bfff640 1 -- 172.21.15.16:0/3004686432 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9588059420 con 0x7f9594144b20 2024-04-24T23:50:42.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.900+0000 7f959a06b640 1 -- 172.21.15.16:0/3004686432 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f959404f5e0 con 0x7f9594144b20 2024-04-24T23:50:42.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.901+0000 7f957bfff640 1 -- 172.21.15.16:0/3004686432 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5184 (secure 0 0 0) 0x7f9588058d20 con 0x7f9594144b20 2024-04-24T23:50:42.902 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:42.902 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5299,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/448535992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":448535992},{"type":"v1","addr":"172.21.15.16:6835","nonce":448535992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5305,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1890687780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1890687780},{"type":"v1","addr":"172.21.15.156:6836","nonce":1890687780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5308,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2538147395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":2538147395},{"type":"v1","addr":"172.21.15.156:6837","nonce":2538147395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-24T23:50:31.671778+0000","modified":"2024-04-24T23:50:35.722577+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5736},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5736":{"gid":5736,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":4,"addr":"172.21.15.16:6834/3643102469","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3643102469},{"type":"v1","addr":"172.21.15.16:6834","nonce":3643102469}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":5736,"qdb_cluster":[5736]},"id":2}]} 2024-04-24T23:50:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 23 2024-04-24T23:50:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.903+0000 7f959a06b640 1 -- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f958006be30 msgr2=0x7f958006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.903+0000 7f959a06b640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f958006be30 0x7f958006e2f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9584000f60 tx=0x7f9584009040 comp rx=0 tx=0).stop 2024-04-24T23:50:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.903+0000 7f959a06b640 1 -- 172.21.15.16:0/3004686432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9594144b20 msgr2=0x7f95941300b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:42.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.903+0000 7f959a06b640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9594144b20 0x7f95941300b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f958800c3c0 tx=0x7f958800c880 comp rx=0 tx=0).stop 2024-04-24T23:50:42.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.903+0000 7f959a06b640 1 -- 172.21.15.16:0/3004686432 shutdown_connections 2024-04-24T23:50:42.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.903+0000 7f959a06b640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f958006be30 0x7f958006e2f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.903+0000 7f959a06b640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f95941457b0 0x7f95940cb250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.903+0000 7f959a06b640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9594144b20 0x7f95941300b0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.903+0000 7f959a06b640 1 --2- 172.21.15.16:0/3004686432 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f95940ced90 0x7f9594133e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:42.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.903+0000 7f959a06b640 1 -- 172.21.15.16:0/3004686432 >> 172.21.15.16:0/3004686432 conn(0x7f95940b2000 msgr2=0x7f959406da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:42.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.903+0000 7f959a06b640 1 -- 172.21.15.16:0/3004686432 shutdown_connections 2024-04-24T23:50:42.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:42.903+0000 7f959a06b640 1 -- 172.21.15.16:0/3004686432 wait complete. 2024-04-24T23:50:42.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:50:43.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.015+0000 7f4604658640 1 Processor -- start 2024-04-24T23:50:43.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.015+0000 7f4604658640 1 -- start start 2024-04-24T23:50:43.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.016+0000 7f4604658640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f45fc1457b0 0x7f45fc0ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.015 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.016+0000 7f4604658640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45fc15f990 0x7f45fc0cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.016+0000 7f4604658640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45fc15fe90 con 0x7f45fc144b20 2024-04-24T23:50:43.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.016+0000 7f4604658640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f45fc06ea50 con 0x7f45fc1457b0 2024-04-24T23:50:43.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.016+0000 7f4604658640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f45fc160170 con 0x7f45fc15f990 2024-04-24T23:50:43.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.016+0000 7f4601bf3640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f45fc1457b0 0x7f45fc0ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.016+0000 7f4602bf5640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45fc15f990 0x7f45fc0cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.016+0000 7f4601bf3640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f45fc1457b0 0x7f45fc0ce680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.16:35270/0 (socket says 172.21.15.16:35270) 2024-04-24T23:50:43.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.016+0000 7f4601bf3640 1 -- 172.21.15.16:0/3436466347 learned_addr learned my addr 172.21.15.16:0/3436466347 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:43.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.016+0000 7f46023f4640 1 --1- 172.21.15.16:0/3436466347 >> v1:172.21.15.156:6789/0 conn(0x7f45fc144b20 0x7f45fc144f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.156:6789/0 says I am v1:172.21.15.16:53552/0 (socket says 172.21.15.16:53552) 2024-04-24T23:50:43.017 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.017+0000 7f4601bf3640 1 -- 172.21.15.16:0/3436466347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45fc15f990 msgr2=0x7f45fc0cebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.017+0000 7f4601bf3640 1 --2- 172.21.15.16:0/3436466347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45fc15f990 0x7f45fc0cebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.017+0000 7f4601bf3640 1 -- 172.21.15.16:0/3436466347 >> v1:172.21.15.156:6789/0 conn(0x7f45fc144b20 legacy=0x7f45fc144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.017+0000 7f4601bf3640 1 -- 172.21.15.16:0/3436466347 --> v2:172.21.15.156:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45fc160450 con 0x7f45fc1457b0 2024-04-24T23:50:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.017+0000 7f46013f2640 1 -- 172.21.15.16:0/3436466347 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2717574940 0 0) 0x7f45fc15fe90 con 0x7f45fc144b20 2024-04-24T23:50:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.017+0000 7f4602bf5640 1 --2- 172.21.15.16:0/3436466347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45fc15f990 0x7f45fc0cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:50:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.017+0000 7f4601bf3640 1 --2- 172.21.15.16:0/3436466347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45fc1457b0 0x7f45fc0ce680 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f45f0009870 tx=0x7f45f002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0c2365fb9fbc85 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.018+0000 7f46013f2640 1 -- 172.21.15.16:0/3436466347 <== mon.1 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f0037070 con 0x7f45fc1457b0 2024-04-24T23:50:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.018+0000 7f46013f2640 1 -- 172.21.15.16:0/3436466347 <== mon.1 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45f0032440 con 0x7f45fc1457b0 2024-04-24T23:50:43.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.018+0000 7f46013f2640 1 -- 172.21.15.16:0/3436466347 <== mon.1 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f0032810 con 0x7f45fc1457b0 2024-04-24T23:50:43.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.018+0000 7f4604658640 1 -- 172.21.15.16:0/3436466347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45fc1457b0 msgr2=0x7f45fc0ce680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.018+0000 7f4604658640 1 --2- 172.21.15.16:0/3436466347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45fc1457b0 0x7f45fc0ce680 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f45f0009870 tx=0x7f45f002a1c0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.018+0000 7f4604658640 1 -- 172.21.15.16:0/3436466347 shutdown_connections 2024-04-24T23:50:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.018+0000 7f4604658640 1 --2- 172.21.15.16:0/3436466347 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45fc15f990 0x7f45fc0cebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.018+0000 7f4604658640 1 --2- 172.21.15.16:0/3436466347 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45fc1457b0 0x7f45fc0ce680 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.018+0000 7f4604658640 1 -- 172.21.15.16:0/3436466347 >> 172.21.15.16:0/3436466347 conn(0x7f45fc0b2000 msgr2=0x7f45fc0cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.018+0000 7f4604658640 1 -- 172.21.15.16:0/3436466347 shutdown_connections 2024-04-24T23:50:43.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.018+0000 7f4604658640 1 -- 172.21.15.16:0/3436466347 wait complete. 2024-04-24T23:50:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.019+0000 7f4604658640 1 Processor -- start 2024-04-24T23:50:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.020+0000 7f4604658640 1 -- start start 2024-04-24T23:50:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.020+0000 7f4604658640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45fc144b20 0x7f45fc1354e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.020+0000 7f4604658640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45fc1457b0 0x7f45fc15e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.020+0000 7f4604658640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45fc15f990 0x7f45fc13a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.020+0000 7f4604658640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f45fc0b3390 con 0x7f45fc144b20 2024-04-24T23:50:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.020+0000 7f4604658640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f45fc06d4b0 con 0x7f45fc1457b0 2024-04-24T23:50:43.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.020+0000 7f4604658640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f45fc06da40 con 0x7f45fc15f990 2024-04-24T23:50:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.020+0000 7f46023f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45fc144b20 0x7f45fc1354e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.020+0000 7f46023f4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45fc144b20 0x7f45fc1354e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51030/0 (socket says 172.21.15.16:51030) 2024-04-24T23:50:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.020+0000 7f46023f4640 1 -- 172.21.15.16:0/2202599498 learned_addr learned my addr 172.21.15.16:0/2202599498 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:43.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.021+0000 7f4601bf3640 1 --2- 172.21.15.16:0/2202599498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45fc1457b0 0x7f45fc15e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.021+0000 7f4602bf5640 1 --2- 172.21.15.16:0/2202599498 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45fc15f990 0x7f45fc13a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.021+0000 7f46023f4640 1 -- 172.21.15.16:0/2202599498 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45fc15f990 msgr2=0x7f45fc13a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.021+0000 7f46023f4640 1 --2- 172.21.15.16:0/2202599498 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45fc15f990 0x7f45fc13a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.021+0000 7f46023f4640 1 -- 172.21.15.16:0/2202599498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45fc1457b0 msgr2=0x7f45fc15e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.021+0000 7f46023f4640 1 --2- 172.21.15.16:0/2202599498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45fc1457b0 0x7f45fc15e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.021+0000 7f46023f4640 1 -- 172.21.15.16:0/2202599498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45fc1395a0 con 0x7f45fc144b20 2024-04-24T23:50:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.021+0000 7f46023f4640 1 --2- 172.21.15.16:0/2202599498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45fc144b20 0x7f45fc1354e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f45f400c4d0 tx=0x7f45f400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.021+0000 7f45faffd640 1 -- 172.21.15.16:0/2202599498 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f4018070 con 0x7f45fc144b20 2024-04-24T23:50:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.021+0000 7f45faffd640 1 -- 172.21.15.16:0/2202599498 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45f4007500 con 0x7f45fc144b20 2024-04-24T23:50:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.021+0000 7f4604658640 1 -- 172.21.15.16:0/2202599498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45fc139880 con 0x7f45fc144b20 2024-04-24T23:50:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.021+0000 7f45faffd640 1 -- 172.21.15.16:0/2202599498 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45f4004a20 con 0x7f45fc144b20 2024-04-24T23:50:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.021+0000 7f4604658640 1 -- 172.21.15.16:0/2202599498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45fc139e40 con 0x7f45fc144b20 2024-04-24T23:50:43.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.022+0000 7f4604658640 1 -- 172.21.15.16:0/2202599498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45fc0731f0 con 0x7f45fc144b20 2024-04-24T23:50:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.022+0000 7f45faffd640 1 -- 172.21.15.16:0/2202599498 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f45f401b070 con 0x7f45fc144b20 2024-04-24T23:50:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.023+0000 7f45faffd640 1 --2- 172.21.15.16:0/2202599498 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f45e806be60 0x7f45e806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.023+0000 7f45faffd640 1 -- 172.21.15.16:0/2202599498 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7f45f408a320 con 0x7f45fc144b20 2024-04-24T23:50:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.023+0000 7f4601bf3640 1 --2- 172.21.15.16:0/2202599498 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f45e806be60 0x7f45e806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.024+0000 7f4601bf3640 1 --2- 172.21.15.16:0/2202599498 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f45e806be60 0x7f45e806e320 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f45f0034000 tx=0x7f45f002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:43.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.027+0000 7f45faffd640 1 -- 172.21.15.16:0/2202599498 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f45f4058b60 con 0x7f45fc144b20 2024-04-24T23:50:43.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.214+0000 7f4604658640 1 -- 172.21.15.16:0/2202599498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f45fc04f5e0 con 0x7f45fc144b20 2024-04-24T23:50:43.214 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.215+0000 7f45faffd640 1 -- 172.21.15.16:0/2202599498 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) ==== 74+0+17050 (secure 0 0 0) 0x7f45f4058460 con 0x7f45fc144b20 2024-04-24T23:50:43.215 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:43.215 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":43,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:50:37.287284+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-04-24T23:50:17.026573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-04-24T23:50:17.415963+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.36","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6834/561619268":"2024-04-25T23:50:30.259891+0000","172.21.15.16:6832/561619268":"2024-04-25T23:50:30.259891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:50:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.217+0000 7f4604658640 1 -- 172.21.15.16:0/2202599498 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f45e806be60 msgr2=0x7f45e806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.216 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.217+0000 7f4604658640 1 --2- 172.21.15.16:0/2202599498 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f45e806be60 0x7f45e806e320 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f45f0034000 tx=0x7f45f002e000 comp rx=0 tx=0).stop 2024-04-24T23:50:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.217+0000 7f4604658640 1 -- 172.21.15.16:0/2202599498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45fc144b20 msgr2=0x7f45fc1354e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.217+0000 7f4604658640 1 --2- 172.21.15.16:0/2202599498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45fc144b20 0x7f45fc1354e0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f45f400c4d0 tx=0x7f45f400c990 comp rx=0 tx=0).stop 2024-04-24T23:50:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.217+0000 7f4604658640 1 -- 172.21.15.16:0/2202599498 shutdown_connections 2024-04-24T23:50:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.217+0000 7f4604658640 1 --2- 172.21.15.16:0/2202599498 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f45e806be60 0x7f45e806e320 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.217+0000 7f4604658640 1 --2- 172.21.15.16:0/2202599498 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f45fc15f990 0x7f45fc13a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.217+0000 7f4604658640 1 --2- 172.21.15.16:0/2202599498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f45fc1457b0 0x7f45fc15e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.217+0000 7f4604658640 1 --2- 172.21.15.16:0/2202599498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f45fc144b20 0x7f45fc1354e0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.217+0000 7f4604658640 1 -- 172.21.15.16:0/2202599498 >> 172.21.15.16:0/2202599498 conn(0x7f45fc0b2000 msgr2=0x7f45fc06ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.217+0000 7f4604658640 1 -- 172.21.15.16:0/2202599498 shutdown_connections 2024-04-24T23:50:43.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.217+0000 7f4604658640 1 -- 172.21.15.16:0/2202599498 wait complete. 2024-04-24T23:50:43.228 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T23:50:43.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.331+0000 7f6cbc3ee640 1 Processor -- start 2024-04-24T23:50:43.331 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.331+0000 7f6cbc3ee640 1 -- start start 2024-04-24T23:50:43.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.332+0000 7f6cbc3ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb40729a0 0x7f6cb4164fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.332+0000 7f6cbc3ee640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cb40b3300 con 0x7f6cb415a450 2024-04-24T23:50:43.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.332+0000 7f6cbc3ee640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6cb40b35e0 con 0x7f6cb406e910 2024-04-24T23:50:43.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.332+0000 7f6cbc3ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6cb40b38c0 con 0x7f6cb40729a0 2024-04-24T23:50:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.332+0000 7f6cba18a640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6cb415a450 0x7f6cb40b2a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32952/0 (socket says 172.21.15.16:32952) 2024-04-24T23:50:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.332+0000 7f6cba18a640 1 -- 172.21.15.16:0/1045510234 learned_addr learned my addr 172.21.15.16:0/1045510234 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.332+0000 7f6cba98b640 1 --2- 172.21.15.16:0/1045510234 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb40729a0 0x7f6cb4164fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.332+0000 7f6cb9188640 1 -- 172.21.15.16:0/1045510234 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1079217752 0 0) 0x7f6cb40b3300 con 0x7f6cb415a450 2024-04-24T23:50:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.332+0000 7f6cb9188640 1 -- 172.21.15.16:0/1045510234 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ca0003520 con 0x7f6cb415a450 2024-04-24T23:50:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9188640 1 -- 172.21.15.16:0/1045510234 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3691795239 0 0) 0x7f6ca0003520 con 0x7f6cb415a450 2024-04-24T23:50:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9188640 1 -- 172.21.15.16:0/1045510234 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb40729a0 msgr2=0x7f6cb4164fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9188640 1 --2- 172.21.15.16:0/1045510234 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb40729a0 0x7f6cb4164fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9188640 1 -- 172.21.15.16:0/1045510234 >> v1:172.21.15.156:6789/0 conn(0x7f6cb406e910 legacy=0x7f6cb4072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9188640 1 -- 172.21.15.16:0/1045510234 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb40cc400 con 0x7f6cb415a450 2024-04-24T23:50:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9188640 1 -- 172.21.15.16:0/1045510234 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f6cb0001670 con 0x7f6cb415a450 2024-04-24T23:50:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9188640 1 -- 172.21.15.16:0/1045510234 >> v1:172.21.15.16:6789/0 conn(0x7f6cb415a450 legacy=0x7f6cb40b2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9188640 1 --2- 172.21.15.16:0/1045510234 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb40729a0 0x7f6ca00039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9188640 1 --2- 172.21.15.16:0/1045510234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0003f20 0x7f6ca0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9188640 1 --2- 172.21.15.16:0/1045510234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca0004d40 0x7f6ca00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9188640 1 -- 172.21.15.16:0/1045510234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb40b38c0 con 0x7f6ca0003f20 2024-04-24T23:50:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9188640 1 -- 172.21.15.16:0/1045510234 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca000bad0 con 0x7f6ca0004d40 2024-04-24T23:50:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9188640 1 -- 172.21.15.16:0/1045510234 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca000bcd0 con 0x7f6cb40729a0 2024-04-24T23:50:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9989640 1 --2- 172.21.15.16:0/1045510234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0003f20 0x7f6ca0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cba18a640 1 --2- 172.21.15.16:0/1045510234 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb40729a0 0x7f6ca00039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cba98b640 1 --2- 172.21.15.16:0/1045510234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca0004d40 0x7f6ca00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9989640 1 -- 172.21.15.16:0/1045510234 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb40729a0 msgr2=0x7f6ca00039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9989640 1 --2- 172.21.15.16:0/1045510234 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb40729a0 0x7f6ca00039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9989640 1 -- 172.21.15.16:0/1045510234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca0004d40 msgr2=0x7f6ca00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9989640 1 --2- 172.21.15.16:0/1045510234 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca0004d40 0x7f6ca00051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.333+0000 7f6cb9989640 1 -- 172.21.15.16:0/1045510234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca0003520 con 0x7f6ca0003f20 2024-04-24T23:50:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.334+0000 7f6cb9989640 1 --2- 172.21.15.16:0/1045510234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0003f20 0x7f6ca0004800 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4001860 tx=0x7f6ca0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e817e6e152cc0f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.334+0000 7f6cb9188640 1 -- 172.21.15.16:0/1045510234 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4037070 con 0x7f6ca0003f20 2024-04-24T23:50:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.334+0000 7f6cb9188640 1 -- 172.21.15.16:0/1045510234 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ca402abf0 con 0x7f6ca0003f20 2024-04-24T23:50:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.334+0000 7f6cb9188640 1 -- 172.21.15.16:0/1045510234 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4001430 con 0x7f6ca0003f20 2024-04-24T23:50:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.335+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1045510234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0003f20 msgr2=0x7f6ca0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.335+0000 7f6cbc3ee640 1 --2- 172.21.15.16:0/1045510234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ca0003f20 0x7f6ca0004800 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4001860 tx=0x7f6ca0005830 comp rx=0 tx=0).stop 2024-04-24T23:50:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.335+0000 7f6cba98b640 1 -- 172.21.15.16:0/1045510234 reap_dead start 2024-04-24T23:50:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.335+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1045510234 shutdown_connections 2024-04-24T23:50:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.335+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1045510234 >> 172.21.15.16:0/1045510234 conn(0x7f6cb40b2000 msgr2=0x7f6cb40cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.335+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1045510234 shutdown_connections 2024-04-24T23:50:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.335+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1045510234 wait complete. 2024-04-24T23:50:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.336+0000 7f6cbc3ee640 1 Processor -- start 2024-04-24T23:50:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.336+0000 7f6cbc3ee640 1 -- start start 2024-04-24T23:50:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cbc3ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca0003f20 0x7f6cb414d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cbc3ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb413efa0 0x7f6cb414b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cbc3ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb414bc60 0x7f6cb40ce310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cbc3ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb40cdd90 con 0x7f6cb413efa0 2024-04-24T23:50:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cbc3ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6cb40cdc10 con 0x7f6ca0003f20 2024-04-24T23:50:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cbc3ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6cb4071a00 con 0x7f6cb414bc60 2024-04-24T23:50:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cb9989640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb413efa0 0x7f6cb414b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cb9989640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb413efa0 0x7f6cb414b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51046/0 (socket says 172.21.15.16:51046) 2024-04-24T23:50:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cb9989640 1 -- 172.21.15.16:0/1473480134 learned_addr learned my addr 172.21.15.16:0/1473480134 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cba18a640 1 --2- 172.21.15.16:0/1473480134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca0003f20 0x7f6cb414d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cba98b640 1 --2- 172.21.15.16:0/1473480134 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb414bc60 0x7f6cb40ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cb9989640 1 -- 172.21.15.16:0/1473480134 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb414bc60 msgr2=0x7f6cb40ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cb9989640 1 --2- 172.21.15.16:0/1473480134 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb414bc60 0x7f6cb40ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cb9989640 1 -- 172.21.15.16:0/1473480134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca0003f20 msgr2=0x7f6cb414d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cb9989640 1 --2- 172.21.15.16:0/1473480134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca0003f20 0x7f6cb414d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cb9989640 1 -- 172.21.15.16:0/1473480134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cb4148e70 con 0x7f6cb413efa0 2024-04-24T23:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6cb9989640 1 --2- 172.21.15.16:0/1473480134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb413efa0 0x7f6cb414b720 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4001980 tx=0x7f6ca4001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.337+0000 7f6caaffd640 1 -- 172.21.15.16:0/1473480134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca403f020 con 0x7f6cb413efa0 2024-04-24T23:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.338+0000 7f6caaffd640 1 -- 172.21.15.16:0/1473480134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ca4005270 con 0x7f6cb413efa0 2024-04-24T23:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.338+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1473480134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cb414dd50 con 0x7f6cb413efa0 2024-04-24T23:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.338+0000 7f6caaffd640 1 -- 172.21.15.16:0/1473480134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca4037040 con 0x7f6cb413efa0 2024-04-24T23:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.338+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1473480134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cb41493f0 con 0x7f6cb413efa0 2024-04-24T23:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.338+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1473480134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6cb414c040 con 0x7f6cb413efa0 2024-04-24T23:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.339+0000 7f6caaffd640 1 -- 172.21.15.16:0/1473480134 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6ca4042020 con 0x7f6cb413efa0 2024-04-24T23:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.339+0000 7f6caaffd640 1 --2- 172.21.15.16:0/1473480134 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6c9806be60 0x7f6c9806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.339+0000 7f6caaffd640 1 -- 172.21.15.16:0/1473480134 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 6216+0+0 (secure 0 0 0) 0x7f6ca40aada0 con 0x7f6cb413efa0 2024-04-24T23:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.340+0000 7f6cba18a640 1 --2- 172.21.15.16:0/1473480134 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6c9806be60 0x7f6c9806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.340+0000 7f6cba18a640 1 --2- 172.21.15.16:0/1473480134 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6c9806be60 0x7f6c9806e320 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6cb00014f0 tx=0x7f6cb0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:43.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.344+0000 7f6caaffd640 1 -- 172.21.15.16:0/1473480134 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6ca4049030 con 0x7f6cb413efa0 2024-04-24T23:50:43.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.565+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1473480134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f6cb40731f0 con 0x7f6cb413efa0 2024-04-24T23:50:43.565 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-04-24T23:50:43.565+0000 7fa77c6c1640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T23:50:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.566+0000 7f6caaffd640 1 -- 172.21.15.16:0/1473480134 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) ==== 161+0+0 (secure 0 0 0) 0x7f6ca4079560 con 0x7f6cb413efa0 2024-04-24T23:50:43.566 INFO:teuthology.orchestra.run.smithi016.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T23:50:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.570+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1473480134 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6c9806be60 msgr2=0x7f6c9806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.570+0000 7f6cbc3ee640 1 --2- 172.21.15.16:0/1473480134 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6c9806be60 0x7f6c9806e320 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6cb00014f0 tx=0x7f6cb0009040 comp rx=0 tx=0).stop 2024-04-24T23:50:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.570+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1473480134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb413efa0 msgr2=0x7f6cb414b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.570+0000 7f6cbc3ee640 1 --2- 172.21.15.16:0/1473480134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb413efa0 0x7f6cb414b720 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6ca4001980 tx=0x7f6ca4001230 comp rx=0 tx=0).stop 2024-04-24T23:50:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.570+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1473480134 shutdown_connections 2024-04-24T23:50:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.570+0000 7f6cbc3ee640 1 --2- 172.21.15.16:0/1473480134 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6c9806be60 0x7f6c9806e320 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.570+0000 7f6cbc3ee640 1 --2- 172.21.15.16:0/1473480134 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6cb414bc60 0x7f6cb40ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.570 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.570+0000 7f6cbc3ee640 1 --2- 172.21.15.16:0/1473480134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cb413efa0 0x7f6cb414b720 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.570+0000 7f6cbc3ee640 1 --2- 172.21.15.16:0/1473480134 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ca0003f20 0x7f6cb414d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.570+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1473480134 >> 172.21.15.16:0/1473480134 conn(0x7f6cb40b2000 msgr2=0x7f6cb4148630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.570+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1473480134 shutdown_connections 2024-04-24T23:50:43.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.570+0000 7f6cbc3ee640 1 -- 172.21.15.16:0/1473480134 wait complete. 2024-04-24T23:50:43.571 INFO:tasks.ceph.mds.a.smithi016.stderr: -550> 2024-04-24T23:50:36.270+0000 7f82a5994640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T23:50:43.581 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T23:50:43.596 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2024-04-24T23:50:43.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.688+0000 7f4159c53640 1 Processor -- start 2024-04-24T23:50:43.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.688+0000 7f4159c53640 1 -- start start 2024-04-24T23:50:43.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.689+0000 7f4159c53640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f415415e070 0x7f415406c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.689+0000 7f4159c53640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f415415e450 con 0x7f41540d3050 2024-04-24T23:50:43.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.689+0000 7f4159c53640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41540ce790 con 0x7f41540b6f90 2024-04-24T23:50:43.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.689+0000 7f4159c53640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f41540cf3a0 con 0x7f415415e070 2024-04-24T23:50:43.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.689+0000 7f41537fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f415415e070 0x7f415406c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.689+0000 7f4153fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f41540d3050 0x7f41540cf830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32962/0 (socket says 172.21.15.16:32962) 2024-04-24T23:50:43.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.689+0000 7f4153fff640 1 -- 172.21.15.16:0/1789844475 learned_addr learned my addr 172.21.15.16:0/1789844475 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:43.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.689+0000 7f41537fe640 1 -- 172.21.15.16:0/1789844475 >> v1:172.21.15.156:6789/0 conn(0x7f41540b6f90 legacy=0x7f41540d2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:43.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.689+0000 7f41537fe640 1 -- 172.21.15.16:0/1789844475 >> v1:172.21.15.16:6789/0 conn(0x7f41540d3050 legacy=0x7f41540cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.689+0000 7f41537fe640 1 -- 172.21.15.16:0/1789844475 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41540cea70 con 0x7f415415e070 2024-04-24T23:50:43.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.689+0000 7f41527fc640 1 -- 172.21.15.16:0/1789844475 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1069335109 0 0) 0x7f415415e450 con 0x7f41540d3050 2024-04-24T23:50:43.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.690+0000 7f41537fe640 1 --2- 172.21.15.16:0/1789844475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f415415e070 0x7f415406c330 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f41440097a0 tx=0x7f414402a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ae984b6a696abf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:43.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.690+0000 7f41527fc640 1 -- 172.21.15.16:0/1789844475 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4144037070 con 0x7f415415e070 2024-04-24T23:50:43.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.690+0000 7f41527fc640 1 -- 172.21.15.16:0/1789844475 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f414402ae50 con 0x7f415415e070 2024-04-24T23:50:43.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.690+0000 7f41527fc640 1 -- 172.21.15.16:0/1789844475 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4144032660 con 0x7f415415e070 2024-04-24T23:50:43.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.691+0000 7f4159c53640 1 -- 172.21.15.16:0/1789844475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f415415e070 msgr2=0x7f415406c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.691+0000 7f4159c53640 1 --2- 172.21.15.16:0/1789844475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f415415e070 0x7f415406c330 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f41440097a0 tx=0x7f414402a140 comp rx=0 tx=0).stop 2024-04-24T23:50:43.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.691+0000 7f4159c53640 1 -- 172.21.15.16:0/1789844475 shutdown_connections 2024-04-24T23:50:43.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.691+0000 7f4159c53640 1 --2- 172.21.15.16:0/1789844475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f415415e070 0x7f415406c330 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.691+0000 7f4159c53640 1 -- 172.21.15.16:0/1789844475 >> 172.21.15.16:0/1789844475 conn(0x7f41540b5850 msgr2=0x7f41540b5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:43.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.691+0000 7f4159c53640 1 -- 172.21.15.16:0/1789844475 shutdown_connections 2024-04-24T23:50:43.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.691+0000 7f4159c53640 1 -- 172.21.15.16:0/1789844475 wait complete. 2024-04-24T23:50:43.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.692+0000 7f4159c53640 1 Processor -- start 2024-04-24T23:50:43.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.692+0000 7f4159c53640 1 -- start start 2024-04-24T23:50:43.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4159c53640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f41540b6f90 0x7f415414d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4159c53640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41540d3050 0x7f41540cc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4159c53640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f415415e070 0x7f415415f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4159c53640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f41540ce3b0 con 0x7f415415e070 2024-04-24T23:50:43.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4159c53640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f41540cf030 con 0x7f41540d3050 2024-04-24T23:50:43.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4159c53640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f41540ce530 con 0x7f41540b6f90 2024-04-24T23:50:43.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4153fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f415415e070 0x7f415415f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4153fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f415415e070 0x7f415415f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51080/0 (socket says 172.21.15.16:51080) 2024-04-24T23:50:43.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4153fff640 1 -- 172.21.15.16:0/4079610240 learned_addr learned my addr 172.21.15.16:0/4079610240 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:43.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f41537fe640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f41540b6f90 0x7f415414d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4153fff640 1 -- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f41540b6f90 msgr2=0x7f415414d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4153fff640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f41540b6f90 0x7f415414d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4152ffd640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41540d3050 0x7f41540cc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4153fff640 1 -- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41540d3050 msgr2=0x7f41540cc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:43.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4153fff640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41540d3050 0x7f41540cc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:43.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4153fff640 1 -- 172.21.15.16:0/4079610240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41540cc620 con 0x7f415415e070 2024-04-24T23:50:43.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.693+0000 7f4152ffd640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41540d3050 0x7f41540cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:43.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.694+0000 7f4153fff640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f415415e070 0x7f415415f690 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f414c00c7a0 tx=0x7f414c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:43.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.694+0000 7f413bfff640 1 -- 172.21.15.16:0/4079610240 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c018070 con 0x7f415415e070 2024-04-24T23:50:43.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.694+0000 7f413bfff640 1 -- 172.21.15.16:0/4079610240 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f414c00a640 con 0x7f415415e070 2024-04-24T23:50:43.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.694+0000 7f4159c53640 1 -- 172.21.15.16:0/4079610240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41540cccc0 con 0x7f415415e070 2024-04-24T23:50:43.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.694+0000 7f413bfff640 1 -- 172.21.15.16:0/4079610240 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f414c00aab0 con 0x7f415415e070 2024-04-24T23:50:43.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.694+0000 7f4159c53640 1 -- 172.21.15.16:0/4079610240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41540cd280 con 0x7f415415e070 2024-04-24T23:50:43.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.695+0000 7f4159c53640 1 -- 172.21.15.16:0/4079610240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4154068f90 con 0x7f415415e070 2024-04-24T23:50:43.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.695+0000 7f413bfff640 1 -- 172.21.15.16:0/4079610240 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f414c01b070 con 0x7f415415e070 2024-04-24T23:50:43.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.696+0000 7f413bfff640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f414006be30 0x7f414006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:43.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.696+0000 7f413bfff640 1 -- 172.21.15.16:0/4079610240 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7f414c08a6e0 con 0x7f415415e070 2024-04-24T23:50:43.697 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.696+0000 7f41537fe640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f414006be30 0x7f414006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:43.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.696+0000 7f41537fe640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f414006be30 0x7f414006e2f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4144034000 tx=0x7f414402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:43.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.700+0000 7f413bfff640 1 -- 172.21.15.16:0/4079610240 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f41540cc620 con 0x7f415415e070 2024-04-24T23:50:43.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:43.922+0000 7f4159c53640 1 -- 172.21.15.16:0/4079610240 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f41540d3520 con 0x7f415415e070 2024-04-24T23:50:44.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.568+0000 7f413bfff640 1 -- 172.21.15.16:0/4079610240 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) ==== 104+0+0 (secure 0 0 0) 0x7f414c058e40 con 0x7f415415e070 2024-04-24T23:50:44.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.571+0000 7f4159c53640 1 -- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f414006be30 msgr2=0x7f414006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:44.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.571+0000 7f4159c53640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f414006be30 0x7f414006e2f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4144034000 tx=0x7f414402e000 comp rx=0 tx=0).stop 2024-04-24T23:50:44.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.571+0000 7f4159c53640 1 -- 172.21.15.16:0/4079610240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f415415e070 msgr2=0x7f415415f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:44.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.571+0000 7f4159c53640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f415415e070 0x7f415415f690 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f414c00c7a0 tx=0x7f414c00cc60 comp rx=0 tx=0).stop 2024-04-24T23:50:44.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.571+0000 7f4159c53640 1 -- 172.21.15.16:0/4079610240 shutdown_connections 2024-04-24T23:50:44.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.571+0000 7f4159c53640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f414006be30 0x7f414006e2f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:44.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.571+0000 7f4159c53640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f415415e070 0x7f415415f690 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:44.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.571+0000 7f4159c53640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f41540d3050 0x7f41540cc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:44.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.571+0000 7f4159c53640 1 --2- 172.21.15.16:0/4079610240 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f41540b6f90 0x7f415414d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:44.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.571+0000 7f4159c53640 1 -- 172.21.15.16:0/4079610240 >> 172.21.15.16:0/4079610240 conn(0x7f41540b5850 msgr2=0x7f41540b7410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:44.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.571+0000 7f4159c53640 1 -- 172.21.15.16:0/4079610240 shutdown_connections 2024-04-24T23:50:44.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.571+0000 7f4159c53640 1 -- 172.21.15.16:0/4079610240 wait complete. 2024-04-24T23:50:44.585 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-24T23:50:44.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.739+0000 7f91af5ee640 1 Processor -- start 2024-04-24T23:50:44.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.739+0000 7f91af5ee640 1 -- start start 2024-04-24T23:50:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.740+0000 7f91af5ee640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91a815df90 0x7f91a8072b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.740+0000 7f91af5ee640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f91a80cfc10 0x7f91a8073050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.740+0000 7f91af5ee640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f91a815e370 con 0x7f91a806f0d0 2024-04-24T23:50:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.740+0000 7f91af5ee640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f91a81627c0 con 0x7f91a815df90 2024-04-24T23:50:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.740+0000 7f91af5ee640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f91a806ec10 con 0x7f91a80cfc10 2024-04-24T23:50:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.740+0000 7f91ad38a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91a815df90 0x7f91a8072b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.740+0000 7f91adb8b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f91a806f0d0 0x7f91a80ad8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32966/0 (socket says 172.21.15.16:32966) 2024-04-24T23:50:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.740+0000 7f91adb8b640 1 -- 172.21.15.16:0/2228590339 learned_addr learned my addr 172.21.15.16:0/2228590339 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.741+0000 7f91acb89640 1 --2- 172.21.15.16:0/2228590339 >> v2:172.21.15.156:3300/0 conn(0x7f91a80cfc10 0x7f91a8073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:44.746 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.741+0000 7f91ad38a640 1 -- 172.21.15.16:0/2228590339 >> v2:172.21.15.156:3300/0 conn(0x7f91a80cfc10 msgr2=0x7f91a8073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.741+0000 7f91ad38a640 1 --2- 172.21.15.16:0/2228590339 >> v2:172.21.15.156:3300/0 conn(0x7f91a80cfc10 0x7f91a8073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.741+0000 7f91ad38a640 1 -- 172.21.15.16:0/2228590339 >> v1:172.21.15.16:6789/0 conn(0x7f91a806f0d0 legacy=0x7f91a80ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.741+0000 7f91ad38a640 1 -- 172.21.15.16:0/2228590339 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a80cf780 con 0x7f91a815df90 2024-04-24T23:50:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.741+0000 7f919ffff640 1 -- 172.21.15.16:0/2228590339 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3892560066 0 0) 0x7f91a815e370 con 0x7f91a806f0d0 2024-04-24T23:50:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.741+0000 7f91acb89640 1 --2- 172.21.15.16:0/2228590339 >> v2:172.21.15.156:3300/0 conn(0x7f91a80cfc10 0x7f91a8073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.741+0000 7f91ad38a640 1 --2- 172.21.15.16:0/2228590339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91a815df90 0x7f91a8072b10 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f91a0009d90 tx=0x7f91a002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25100b706c7e8956 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.741+0000 7f919ffff640 1 -- 172.21.15.16:0/2228590339 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0031600 con 0x7f91a815df90 2024-04-24T23:50:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.742+0000 7f919ffff640 1 -- 172.21.15.16:0/2228590339 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a00317a0 con 0x7f91a815df90 2024-04-24T23:50:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.742+0000 7f919ffff640 1 -- 172.21.15.16:0/2228590339 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0031b70 con 0x7f91a815df90 2024-04-24T23:50:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.742+0000 7f91af5ee640 1 -- 172.21.15.16:0/2228590339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91a815df90 msgr2=0x7f91a8072b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.742+0000 7f91af5ee640 1 --2- 172.21.15.16:0/2228590339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91a815df90 0x7f91a8072b10 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f91a0009d90 tx=0x7f91a002a610 comp rx=0 tx=0).stop 2024-04-24T23:50:44.747 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.742+0000 7f91af5ee640 1 -- 172.21.15.16:0/2228590339 shutdown_connections 2024-04-24T23:50:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.742+0000 7f91af5ee640 1 --2- 172.21.15.16:0/2228590339 >> v2:172.21.15.156:3300/0 conn(0x7f91a80cfc10 0x7f91a8073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.742+0000 7f91af5ee640 1 --2- 172.21.15.16:0/2228590339 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91a815df90 0x7f91a8072b10 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.742+0000 7f91af5ee640 1 -- 172.21.15.16:0/2228590339 >> 172.21.15.16:0/2228590339 conn(0x7f91a80d0aa0 msgr2=0x7f91a80d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.742+0000 7f91af5ee640 1 -- 172.21.15.16:0/2228590339 shutdown_connections 2024-04-24T23:50:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.742+0000 7f91af5ee640 1 -- 172.21.15.16:0/2228590339 wait complete. 2024-04-24T23:50:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.743+0000 7f91af5ee640 1 Processor -- start 2024-04-24T23:50:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.743+0000 7f91af5ee640 1 -- start start 2024-04-24T23:50:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.744+0000 7f91af5ee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91a806f0d0 0x7f91a8147300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.744+0000 7f91af5ee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91a80cfc10 0x7f91a8149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.744+0000 7f91af5ee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f91a815df90 0x7f91a815a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.744+0000 7f91af5ee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f91a8162170 con 0x7f91a806f0d0 2024-04-24T23:50:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.744+0000 7f91af5ee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f91a8161e90 con 0x7f91a80cfc10 2024-04-24T23:50:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.744+0000 7f91af5ee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f91a8162450 con 0x7f91a815df90 2024-04-24T23:50:44.748 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.744+0000 7f91ad38a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91a806f0d0 0x7f91a8147300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.744+0000 7f91ad38a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91a806f0d0 0x7f91a8147300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51104/0 (socket says 172.21.15.16:51104) 2024-04-24T23:50:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.744+0000 7f91ad38a640 1 -- 172.21.15.16:0/1732489935 learned_addr learned my addr 172.21.15.16:0/1732489935 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.744+0000 7f91acb89640 1 --2- 172.21.15.16:0/1732489935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91a80cfc10 0x7f91a8149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.744+0000 7f91ad38a640 1 -- 172.21.15.16:0/1732489935 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f91a815df90 msgr2=0x7f91a815a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.744+0000 7f91ad38a640 1 --2- 172.21.15.16:0/1732489935 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f91a815df90 0x7f91a815a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.745+0000 7f91ad38a640 1 -- 172.21.15.16:0/1732489935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91a80cfc10 msgr2=0x7f91a8149ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.745+0000 7f91ad38a640 1 --2- 172.21.15.16:0/1732489935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91a80cfc10 0x7f91a8149ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.745+0000 7f91ad38a640 1 -- 172.21.15.16:0/1732489935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a8159ee0 con 0x7f91a806f0d0 2024-04-24T23:50:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.745+0000 7f91acb89640 1 --2- 172.21.15.16:0/1732489935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91a80cfc10 0x7f91a8149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.745+0000 7f91ad38a640 1 --2- 172.21.15.16:0/1732489935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91a806f0d0 0x7f91a8147300 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f91a0000f80 tx=0x7f91a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.745+0000 7f919dffb640 1 -- 172.21.15.16:0/1732489935 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0040070 con 0x7f91a806f0d0 2024-04-24T23:50:44.749 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.745+0000 7f919dffb640 1 -- 172.21.15.16:0/1732489935 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91a0030be0 con 0x7f91a806f0d0 2024-04-24T23:50:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.745+0000 7f91af5ee640 1 -- 172.21.15.16:0/1732489935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a814a310 con 0x7f91a806f0d0 2024-04-24T23:50:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.745+0000 7f919dffb640 1 -- 172.21.15.16:0/1732489935 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91a0009450 con 0x7f91a806f0d0 2024-04-24T23:50:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.745+0000 7f91af5ee640 1 -- 172.21.15.16:0/1732489935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a814a8d0 con 0x7f91a806f0d0 2024-04-24T23:50:44.750 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.746+0000 7f91af5ee640 1 -- 172.21.15.16:0/1732489935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91a8068f90 con 0x7f91a806f0d0 2024-04-24T23:50:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.747+0000 7f919dffb640 1 -- 172.21.15.16:0/1732489935 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f91a0039040 con 0x7f91a806f0d0 2024-04-24T23:50:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.747+0000 7f919dffb640 1 --2- 172.21.15.16:0/1732489935 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f918c06be60 0x7f918c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.747+0000 7f919dffb640 1 -- 172.21.15.16:0/1732489935 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 6302+0+0 (secure 0 0 0) 0x7f91a00ac1f0 con 0x7f91a806f0d0 2024-04-24T23:50:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.747+0000 7f91acb89640 1 --2- 172.21.15.16:0/1732489935 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f918c06be60 0x7f918c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.748+0000 7f91acb89640 1 --2- 172.21.15.16:0/1732489935 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f918c06be60 0x7f918c06e320 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9198006fd0 tx=0x7f9198008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:44.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.751+0000 7f919dffb640 1 -- 172.21.15.16:0/1732489935 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f91a007a950 con 0x7f91a806f0d0 2024-04-24T23:50:44.938 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:44.936+0000 7f91af5ee640 1 -- 172.21.15.16:0/1732489935 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f91a80ae8f0 con 0x7f91a806f0d0 2024-04-24T23:50:45.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.572+0000 7f919dffb640 1 -- 172.21.15.16:0/1732489935 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v45) ==== 181+0+0 (secure 0 0 0) 0x7f91a007a250 con 0x7f91a806f0d0 2024-04-24T23:50:45.572 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' removed 2024-04-24T23:50:45.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.576+0000 7f91af5ee640 1 -- 172.21.15.16:0/1732489935 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f918c06be60 msgr2=0x7f918c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:45.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.576+0000 7f91af5ee640 1 --2- 172.21.15.16:0/1732489935 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f918c06be60 0x7f918c06e320 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9198006fd0 tx=0x7f9198008040 comp rx=0 tx=0).stop 2024-04-24T23:50:45.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.576+0000 7f91af5ee640 1 -- 172.21.15.16:0/1732489935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91a806f0d0 msgr2=0x7f91a8147300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:45.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.576+0000 7f91af5ee640 1 --2- 172.21.15.16:0/1732489935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91a806f0d0 0x7f91a8147300 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f91a0000f80 tx=0x7f91a0004a40 comp rx=0 tx=0).stop 2024-04-24T23:50:45.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.576+0000 7f91af5ee640 1 -- 172.21.15.16:0/1732489935 shutdown_connections 2024-04-24T23:50:45.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.576+0000 7f91af5ee640 1 --2- 172.21.15.16:0/1732489935 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f918c06be60 0x7f918c06e320 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:45.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.576+0000 7f91af5ee640 1 --2- 172.21.15.16:0/1732489935 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f91a815df90 0x7f91a815a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:45.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.576+0000 7f91af5ee640 1 --2- 172.21.15.16:0/1732489935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f91a80cfc10 0x7f91a8149ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:45.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.576+0000 7f91af5ee640 1 --2- 172.21.15.16:0/1732489935 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91a806f0d0 0x7f91a8147300 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:45.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.576+0000 7f91af5ee640 1 -- 172.21.15.16:0/1732489935 >> 172.21.15.16:0/1732489935 conn(0x7f91a80d0aa0 msgr2=0x7f91a80b1370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:45.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.576+0000 7f91af5ee640 1 -- 172.21.15.16:0/1732489935 shutdown_connections 2024-04-24T23:50:45.576 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.576+0000 7f91af5ee640 1 -- 172.21.15.16:0/1732489935 wait complete. 2024-04-24T23:50:45.589 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-24T23:50:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.700+0000 7f9a5aff4640 1 Processor -- start 2024-04-24T23:50:45.700 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.700+0000 7f9a5aff4640 1 -- start start 2024-04-24T23:50:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.701+0000 7f9a5aff4640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9a5406bcb0 0x7f9a5406c110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.701+0000 7f9a5aff4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a540d2260 0x7f9a540d26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.701+0000 7f9a5aff4640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a540d1aa0 con 0x7f9a5415dfb0 2024-04-24T23:50:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.701+0000 7f9a5aff4640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9a540d02b0 con 0x7f9a540d2260 2024-04-24T23:50:45.701 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.701+0000 7f9a5aff4640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f9a5406c650 con 0x7f9a5406bcb0 2024-04-24T23:50:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.701+0000 7f9a59591640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a540d2260 0x7f9a540d26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.701+0000 7f9a58d90640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9a5415dfb0 0x7f9a540cfba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32976/0 (socket says 172.21.15.16:32976) 2024-04-24T23:50:45.702 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.701+0000 7f9a58d90640 1 -- 172.21.15.16:0/644819772 learned_addr learned my addr 172.21.15.16:0/644819772 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.702+0000 7f9a53fff640 1 --2- 172.21.15.16:0/644819772 >> v2:172.21.15.156:3300/0 conn(0x7f9a5406bcb0 0x7f9a5406c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.702+0000 7f9a59591640 1 -- 172.21.15.16:0/644819772 >> v2:172.21.15.156:3300/0 conn(0x7f9a5406bcb0 msgr2=0x7f9a5406c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.702+0000 7f9a59591640 1 --2- 172.21.15.16:0/644819772 >> v2:172.21.15.156:3300/0 conn(0x7f9a5406bcb0 0x7f9a5406c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.702+0000 7f9a59591640 1 -- 172.21.15.16:0/644819772 >> v1:172.21.15.16:6789/0 conn(0x7f9a5415dfb0 legacy=0x7f9a540cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:45.704 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.702+0000 7f9a59591640 1 -- 172.21.15.16:0/644819772 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a5406b470 con 0x7f9a540d2260 2024-04-24T23:50:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.702+0000 7f9a537fe640 1 -- 172.21.15.16:0/644819772 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 331987226 0 0) 0x7f9a540d1aa0 con 0x7f9a5415dfb0 2024-04-24T23:50:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.702+0000 7f9a59591640 1 --2- 172.21.15.16:0/644819772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a540d2260 0x7f9a540d26b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9a540d4e90 tx=0x7f9a4402a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04b0bac3cae1bec server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.703+0000 7f9a537fe640 1 -- 172.21.15.16:0/644819772 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a44037070 con 0x7f9a540d2260 2024-04-24T23:50:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.703+0000 7f9a537fe640 1 -- 172.21.15.16:0/644819772 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a44032440 con 0x7f9a540d2260 2024-04-24T23:50:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.703+0000 7f9a537fe640 1 -- 172.21.15.16:0/644819772 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a44032810 con 0x7f9a540d2260 2024-04-24T23:50:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.703+0000 7f9a5aff4640 1 -- 172.21.15.16:0/644819772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a540d2260 msgr2=0x7f9a540d26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.703+0000 7f9a5aff4640 1 --2- 172.21.15.16:0/644819772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a540d2260 0x7f9a540d26b0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f9a540d4e90 tx=0x7f9a4402a1a0 comp rx=0 tx=0).stop 2024-04-24T23:50:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.703+0000 7f9a5aff4640 1 -- 172.21.15.16:0/644819772 shutdown_connections 2024-04-24T23:50:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.703+0000 7f9a5aff4640 1 --2- 172.21.15.16:0/644819772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a540d2260 0x7f9a540d26b0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.703+0000 7f9a5aff4640 1 --2- 172.21.15.16:0/644819772 >> v2:172.21.15.156:3300/0 conn(0x7f9a5406bcb0 0x7f9a5406c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.703+0000 7f9a5aff4640 1 -- 172.21.15.16:0/644819772 >> 172.21.15.16:0/644819772 conn(0x7f9a540d3e30 msgr2=0x7f9a540d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:45.705 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.703+0000 7f9a5aff4640 1 -- 172.21.15.16:0/644819772 shutdown_connections 2024-04-24T23:50:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.703+0000 7f9a5aff4640 1 -- 172.21.15.16:0/644819772 wait complete. 2024-04-24T23:50:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.704+0000 7f9a5aff4640 1 Processor -- start 2024-04-24T23:50:45.706 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.704+0000 7f9a5aff4640 1 -- start start 2024-04-24T23:50:45.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.705+0000 7f9a5aff4640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a5406bcb0 0x7f9a540caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:45.708 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.705+0000 7f9a5aff4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a540d2260 0x7f9a54150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.705+0000 7f9a5aff4640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a5415dfb0 0x7f9a54160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.705+0000 7f9a5aff4640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9a5415e390 con 0x7f9a540d2260 2024-04-24T23:50:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.705+0000 7f9a5aff4640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9a540d1dd0 con 0x7f9a5415dfb0 2024-04-24T23:50:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.705+0000 7f9a5aff4640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f9a5406ba10 con 0x7f9a5406bcb0 2024-04-24T23:50:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.705+0000 7f9a53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a540d2260 0x7f9a54150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.705+0000 7f9a53fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a540d2260 0x7f9a54150220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51128/0 (socket says 172.21.15.16:51128) 2024-04-24T23:50:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a53fff640 1 -- 172.21.15.16:0/2771436344 learned_addr learned my addr 172.21.15.16:0/2771436344 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a58d90640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a5406bcb0 0x7f9a540caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a59591640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a5415dfb0 0x7f9a54160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a53fff640 1 -- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a5406bcb0 msgr2=0x7f9a540caf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a53fff640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a5406bcb0 0x7f9a540caf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:45.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a53fff640 1 -- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a5415dfb0 msgr2=0x7f9a54160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a53fff640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a5415dfb0 0x7f9a54160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a53fff640 1 -- 172.21.15.16:0/2771436344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a54161400 con 0x7f9a540d2260 2024-04-24T23:50:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a59591640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a5415dfb0 0x7f9a54160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a53fff640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a540d2260 0x7f9a54150220 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800d620 tx=0x7f9a4800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a517fa640 1 -- 172.21.15.16:0/2771436344 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4800b620 con 0x7f9a540d2260 2024-04-24T23:50:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a517fa640 1 -- 172.21.15.16:0/2771436344 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a48019070 con 0x7f9a540d2260 2024-04-24T23:50:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a5aff4640 1 -- 172.21.15.16:0/2771436344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a540cb4e0 con 0x7f9a540d2260 2024-04-24T23:50:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a517fa640 1 -- 172.21.15.16:0/2771436344 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a4800ba00 con 0x7f9a540d2260 2024-04-24T23:50:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.706+0000 7f9a5aff4640 1 -- 172.21.15.16:0/2771436344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a540cd950 con 0x7f9a540d2260 2024-04-24T23:50:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.707+0000 7f9a5aff4640 1 -- 172.21.15.16:0/2771436344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a54068f90 con 0x7f9a540d2260 2024-04-24T23:50:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.708+0000 7f9a517fa640 1 -- 172.21.15.16:0/2771436344 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9a4800bba0 con 0x7f9a540d2260 2024-04-24T23:50:45.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.708+0000 7f9a517fa640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9a4006be30 0x7f9a4006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:45.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.708+0000 7f9a517fa640 1 -- 172.21.15.16:0/2771436344 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5864+0+0 (secure 0 0 0) 0x7f9a4808b270 con 0x7f9a540d2260 2024-04-24T23:50:45.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.708+0000 7f9a58d90640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9a4006be30 0x7f9a4006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:45.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.709+0000 7f9a58d90640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9a4006be30 0x7f9a4006e2f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c0012e0 tx=0x7f9a4c001700 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.712+0000 7f9a517fa640 1 -- 172.21.15.16:0/2771436344 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9a48059b80 con 0x7f9a540d2260 2024-04-24T23:50:45.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:45.897+0000 7f9a5aff4640 1 -- 172.21.15.16:0/2771436344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f9a540cecb0 con 0x7f9a540d2260 2024-04-24T23:50:46.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.578+0000 7f9a517fa640 1 -- 172.21.15.16:0/2771436344 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v46) ==== 169+0+0 (secure 0 0 0) 0x7f9a48059480 con 0x7f9a540d2260 2024-04-24T23:50:46.578 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' removed 2024-04-24T23:50:46.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.583+0000 7f9a5aff4640 1 -- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9a4006be30 msgr2=0x7f9a4006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:46.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.583+0000 7f9a5aff4640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9a4006be30 0x7f9a4006e2f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9a4c0012e0 tx=0x7f9a4c001700 comp rx=0 tx=0).stop 2024-04-24T23:50:46.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.583+0000 7f9a5aff4640 1 -- 172.21.15.16:0/2771436344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a540d2260 msgr2=0x7f9a54150220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:46.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.584+0000 7f9a5aff4640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a540d2260 0x7f9a54150220 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f9a4800d620 tx=0x7f9a4800dae0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.587+0000 7f9a5aff4640 1 -- 172.21.15.16:0/2771436344 shutdown_connections 2024-04-24T23:50:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.587+0000 7f9a5aff4640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9a4006be30 0x7f9a4006e2f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.587+0000 7f9a5aff4640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9a5415dfb0 0x7f9a54160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.587+0000 7f9a5aff4640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a540d2260 0x7f9a54150220 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.587+0000 7f9a5aff4640 1 --2- 172.21.15.16:0/2771436344 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9a5406bcb0 0x7f9a540caf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.587+0000 7f9a5aff4640 1 -- 172.21.15.16:0/2771436344 >> 172.21.15.16:0/2771436344 conn(0x7f9a540d3e30 msgr2=0x7f9a540b8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.587+0000 7f9a5aff4640 1 -- 172.21.15.16:0/2771436344 shutdown_connections 2024-04-24T23:50:46.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.588+0000 7f9a5aff4640 1 -- 172.21.15.16:0/2771436344 wait complete. 2024-04-24T23:50:46.604 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-04-24T23:50:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.713+0000 7f6ff4529640 1 Processor -- start 2024-04-24T23:50:46.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.713+0000 7f6ff4529640 1 -- start start 2024-04-24T23:50:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.714+0000 7f6ff4529640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f6fec0b85d0 0x7f6fec168e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.714+0000 7f6ff4529640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fec15e6d0 con 0x7f6fec0d26e0 2024-04-24T23:50:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.714+0000 7f6ff4529640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fec06c230 con 0x7f6fec15e2f0 2024-04-24T23:50:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.714+0000 7f6ff4529640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f6fec0d2ca0 con 0x7f6fec0b85d0 2024-04-24T23:50:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.714+0000 7f6ff1ac4640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6fec0d26e0 0x7f6fec06bb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57614/0 (socket says 172.21.15.16:57614) 2024-04-24T23:50:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.714+0000 7f6ff1ac4640 1 -- 172.21.15.16:0/2869473422 learned_addr learned my addr 172.21.15.16:0/2869473422 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff2ac6640 1 --2- 172.21.15.16:0/2869473422 >> v2:172.21.15.156:3300/0 conn(0x7f6fec0b85d0 0x7f6fec168e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 -- 172.21.15.16:0/2869473422 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3641043499 0 0) 0x7f6fec15e6d0 con 0x7f6fec0d26e0 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 -- 172.21.15.16:0/2869473422 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fd8003520 con 0x7f6fec0d26e0 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 -- 172.21.15.16:0/2869473422 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 862637711 0 0) 0x7f6fd8003520 con 0x7f6fec0d26e0 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 -- 172.21.15.16:0/2869473422 >> v2:172.21.15.156:3300/0 conn(0x7f6fec0b85d0 msgr2=0x7f6fec168e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 --2- 172.21.15.16:0/2869473422 >> v2:172.21.15.156:3300/0 conn(0x7f6fec0b85d0 0x7f6fec168e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 -- 172.21.15.16:0/2869473422 >> v1:172.21.15.156:6789/0 conn(0x7f6fec15e2f0 legacy=0x7f6fec06b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 -- 172.21.15.16:0/2869473422 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fec0d2f80 con 0x7f6fec0d26e0 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 -- 172.21.15.16:0/2869473422 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f6fdc001670 con 0x7f6fec0d26e0 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 -- 172.21.15.16:0/2869473422 >> v1:172.21.15.16:6789/0 conn(0x7f6fec0d26e0 legacy=0x7f6fec06bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 --2- 172.21.15.16:0/2869473422 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fd80039b0 0x7f6fd8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 --2- 172.21.15.16:0/2869473422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8004800 0x7f6fd8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 --2- 172.21.15.16:0/2869473422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fd80051c0 0x7f6fd800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 -- 172.21.15.16:0/2869473422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec0d2ca0 con 0x7f6fd8004800 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 -- 172.21.15.16:0/2869473422 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6fd800c060 con 0x7f6fd80051c0 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff12c3640 1 -- 172.21.15.16:0/2869473422 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6fd800c260 con 0x7f6fd80039b0 2024-04-24T23:50:46.717 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff1ac4640 1 --2- 172.21.15.16:0/2869473422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8004800 0x7f6fd8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff2ac6640 1 --2- 172.21.15.16:0/2869473422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fd80051c0 0x7f6fd800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff22c5640 1 --2- 172.21.15.16:0/2869473422 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fd80039b0 0x7f6fd8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff1ac4640 1 -- 172.21.15.16:0/2869473422 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fd80039b0 msgr2=0x7f6fd8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff1ac4640 1 --2- 172.21.15.16:0/2869473422 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fd80039b0 0x7f6fd8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff1ac4640 1 -- 172.21.15.16:0/2869473422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fd80051c0 msgr2=0x7f6fd800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff1ac4640 1 --2- 172.21.15.16:0/2869473422 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fd80051c0 0x7f6fd800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff2ac6640 1 -- 172.21.15.16:0/2869473422 reap_dead start 2024-04-24T23:50:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.715+0000 7f6ff1ac4640 1 -- 172.21.15.16:0/2869473422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd8003520 con 0x7f6fd8004800 2024-04-24T23:50:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.716+0000 7f6ff1ac4640 1 --2- 172.21.15.16:0/2869473422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8004800 0x7f6fd8004c80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc00c240 tx=0x7f6fd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6827d7bcfeebc6b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.716+0000 7f6ff12c3640 1 -- 172.21.15.16:0/2869473422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc007cf0 con 0x7f6fd8004800 2024-04-24T23:50:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.716+0000 7f6ff12c3640 1 -- 172.21.15.16:0/2869473422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fdc005af0 con 0x7f6fd8004800 2024-04-24T23:50:46.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.716+0000 7f6ff12c3640 1 -- 172.21.15.16:0/2869473422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fdc007070 con 0x7f6fd8004800 2024-04-24T23:50:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.717+0000 7f6ff4529640 1 -- 172.21.15.16:0/2869473422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8004800 msgr2=0x7f6fd8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.717+0000 7f6ff4529640 1 --2- 172.21.15.16:0/2869473422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8004800 0x7f6fd8004c80 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc00c240 tx=0x7f6fd8005830 comp rx=0 tx=0).stop 2024-04-24T23:50:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.717+0000 7f6ff4529640 1 -- 172.21.15.16:0/2869473422 shutdown_connections 2024-04-24T23:50:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.717+0000 7f6ff4529640 1 --2- 172.21.15.16:0/2869473422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8004800 0x7f6fd8004c80 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.717+0000 7f6ff4529640 1 -- 172.21.15.16:0/2869473422 >> 172.21.15.16:0/2869473422 conn(0x7f6fec0d4850 msgr2=0x7f6fec0d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.717+0000 7f6ff4529640 1 -- 172.21.15.16:0/2869473422 shutdown_connections 2024-04-24T23:50:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.717+0000 7f6ff4529640 1 -- 172.21.15.16:0/2869473422 wait complete. 2024-04-24T23:50:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.718+0000 7f6ff4529640 1 Processor -- start 2024-04-24T23:50:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.718+0000 7f6ff4529640 1 -- start start 2024-04-24T23:50:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff4529640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8004800 0x7f6fec1f81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff4529640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fec15be80 0x7f6fec0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff4529640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fec0cbbd0 0x7f6fec0c9f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff4529640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec0b70a0 con 0x7f6fd8004800 2024-04-24T23:50:46.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff4529640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6fec0d3540 con 0x7f6fec0cbbd0 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff4529640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f6fec06b410 con 0x7f6fec15be80 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff22c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8004800 0x7f6fec1f81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff22c5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8004800 0x7f6fec1f81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48858/0 (socket says 172.21.15.16:48858) 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff22c5640 1 -- 172.21.15.16:0/582305861 learned_addr learned my addr 172.21.15.16:0/582305861 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff22c5640 1 -- 172.21.15.16:0/582305861 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fec15be80 msgr2=0x7f6fec0c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff1ac4640 1 --2- 172.21.15.16:0/582305861 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fec15be80 0x7f6fec0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff2ac6640 1 --2- 172.21.15.16:0/582305861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fec0cbbd0 0x7f6fec0c9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff22c5640 1 --2- 172.21.15.16:0/582305861 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fec15be80 0x7f6fec0c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff22c5640 1 -- 172.21.15.16:0/582305861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fec0cbbd0 msgr2=0x7f6fec0c9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff22c5640 1 --2- 172.21.15.16:0/582305861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fec0cbbd0 0x7f6fec0c9f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.719+0000 7f6ff22c5640 1 -- 172.21.15.16:0/582305861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fec0ab4e0 con 0x7f6fd8004800 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.720+0000 7f6ff22c5640 1 --2- 172.21.15.16:0/582305861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8004800 0x7f6fec1f81f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6fe800fbf0 tx=0x7f6fe800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.720+0000 7f6fe2ffd640 1 -- 172.21.15.16:0/582305861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe800dbd0 con 0x7f6fd8004800 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.720+0000 7f6fe2ffd640 1 -- 172.21.15.16:0/582305861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fe8018070 con 0x7f6fd8004800 2024-04-24T23:50:46.723 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.720+0000 7f6ff4529640 1 -- 172.21.15.16:0/582305861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fec0cae10 con 0x7f6fd8004800 2024-04-24T23:50:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.720+0000 7f6fe2ffd640 1 -- 172.21.15.16:0/582305861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8003970 con 0x7f6fd8004800 2024-04-24T23:50:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.720+0000 7f6ff4529640 1 -- 172.21.15.16:0/582305861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fec0cb3d0 con 0x7f6fd8004800 2024-04-24T23:50:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.721+0000 7f6ff4529640 1 -- 172.21.15.16:0/582305861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fec068f90 con 0x7f6fd8004800 2024-04-24T23:50:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.721+0000 7f6fe2ffd640 1 -- 172.21.15.16:0/582305861 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6fe8003b10 con 0x7f6fd8004800 2024-04-24T23:50:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.722+0000 7f6fe2ffd640 1 --2- 172.21.15.16:0/582305861 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6fd006be60 0x7f6fd006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.722+0000 7f6fe2ffd640 1 -- 172.21.15.16:0/582305861 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f6fe808ae00 con 0x7f6fd8004800 2024-04-24T23:50:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.722+0000 7f6ff1ac4640 1 --2- 172.21.15.16:0/582305861 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6fd006be60 0x7f6fd006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:46.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.722+0000 7f6ff1ac4640 1 --2- 172.21.15.16:0/582305861 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6fd006be60 0x7f6fd006e320 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc032000 tx=0x7f6fdc009030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.726+0000 7f6fe2ffd640 1 -- 172.21.15.16:0/582305861 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f6fe80598d0 con 0x7f6fd8004800 2024-04-24T23:50:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.915+0000 7f6ff4529640 1 -- 172.21.15.16:0/582305861 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f6fec06cb30 con 0x7f6fd8004800 2024-04-24T23:50:46.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.918+0000 7f6fe2ffd640 1 -- 172.21.15.16:0/582305861 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f6fe80591d0 con 0x7f6fd8004800 2024-04-24T23:50:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.920+0000 7f6ff4529640 1 -- 172.21.15.16:0/582305861 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6fd006be60 msgr2=0x7f6fd006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.920+0000 7f6ff4529640 1 --2- 172.21.15.16:0/582305861 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6fd006be60 0x7f6fd006e320 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f6fdc032000 tx=0x7f6fdc009030 comp rx=0 tx=0).stop 2024-04-24T23:50:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.920+0000 7f6ff4529640 1 -- 172.21.15.16:0/582305861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8004800 msgr2=0x7f6fec1f81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.920+0000 7f6ff4529640 1 --2- 172.21.15.16:0/582305861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8004800 0x7f6fec1f81f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6fe800fbf0 tx=0x7f6fe800d520 comp rx=0 tx=0).stop 2024-04-24T23:50:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.920+0000 7f6ff4529640 1 -- 172.21.15.16:0/582305861 shutdown_connections 2024-04-24T23:50:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.920+0000 7f6ff4529640 1 --2- 172.21.15.16:0/582305861 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f6fd006be60 0x7f6fd006e320 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.920+0000 7f6ff4529640 1 --2- 172.21.15.16:0/582305861 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6fec0cbbd0 0x7f6fec0c9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.920+0000 7f6ff4529640 1 --2- 172.21.15.16:0/582305861 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f6fec15be80 0x7f6fec0c9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.920+0000 7f6ff4529640 1 --2- 172.21.15.16:0/582305861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6fd8004800 0x7f6fec1f81f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:46.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.920+0000 7f6ff4529640 1 -- 172.21.15.16:0/582305861 >> 172.21.15.16:0/582305861 conn(0x7f6fec0d4850 msgr2=0x7f6fec0b89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:46.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.920+0000 7f6ff4529640 1 -- 172.21.15.16:0/582305861 shutdown_connections 2024-04-24T23:50:46.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:46.920+0000 7f6ff4529640 1 -- 172.21.15.16:0/582305861 wait complete. 2024-04-24T23:50:46.931 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_recovery_fs.TestFSRecovery.test_recover_fs_after_fsmap_removal' 2024-04-24T23:50:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.032+0000 7feb6ad30640 1 Processor -- start 2024-04-24T23:50:47.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.032+0000 7feb6ad30640 1 -- start start 2024-04-24T23:50:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.033+0000 7feb6ad30640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7feb6413eba0 0x7feb64071a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.033+0000 7feb6ad30640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb64071f40 0x7feb6406ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.033+0000 7feb6ad30640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb6406bf40 0x7feb641413c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.033+0000 7feb6ad30640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7feb64160480 con 0x7feb64071f40 2024-04-24T23:50:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.033+0000 7feb6ad30640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7feb6415f890 con 0x7feb6413eba0 2024-04-24T23:50:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.033+0000 7feb6ad30640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7feb6415fb50 con 0x7feb6406bf40 2024-04-24T23:50:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.033+0000 7feb63fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb64071f40 0x7feb6406ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.033+0000 7feb63fff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feb64071f40 0x7feb6406ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48870/0 (socket says 172.21.15.16:48870) 2024-04-24T23:50:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.033+0000 7feb63fff640 1 -- 172.21.15.16:0/3813836442 learned_addr learned my addr 172.21.15.16:0/3813836442 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.033+0000 7feb63fff640 1 -- 172.21.15.16:0/3813836442 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb6406bf40 msgr2=0x7feb641413c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:47.034 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.033+0000 7feb63fff640 1 --2- 172.21.15.16:0/3813836442 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb6406bf40 0x7feb641413c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.033+0000 7feb63fff640 1 -- 172.21.15.16:0/3813836442 >> v2:172.21.15.156:3300/0 conn(0x7feb6413eba0 msgr2=0x7feb64071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.033+0000 7feb63fff640 1 --2- 172.21.15.16:0/3813836442 >> v2:172.21.15.156:3300/0 conn(0x7feb6413eba0 0x7feb64071a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.033+0000 7feb63fff640 1 -- 172.21.15.16:0/3813836442 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6415fe30 con 0x7feb64071f40 2024-04-24T23:50:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.034+0000 7feb63fff640 1 --2- 172.21.15.16:0/3813836442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb64071f40 0x7feb6406ba00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7feb64072ac0 tx=0x7feb5002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3478bf2a74babae8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.034+0000 7feb637fe640 1 -- 172.21.15.16:0/3813836442 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb50037070 con 0x7feb64071f40 2024-04-24T23:50:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.034+0000 7feb637fe640 1 -- 172.21.15.16:0/3813836442 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb50032440 con 0x7feb64071f40 2024-04-24T23:50:47.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.034+0000 7feb637fe640 1 -- 172.21.15.16:0/3813836442 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb50032810 con 0x7feb64071f40 2024-04-24T23:50:47.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.035+0000 7feb6ad30640 1 -- 172.21.15.16:0/3813836442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb64071f40 msgr2=0x7feb6406ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:47.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.035+0000 7feb6ad30640 1 --2- 172.21.15.16:0/3813836442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb64071f40 0x7feb6406ba00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7feb64072ac0 tx=0x7feb5002a1e0 comp rx=0 tx=0).stop 2024-04-24T23:50:47.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.035+0000 7feb6ad30640 1 -- 172.21.15.16:0/3813836442 shutdown_connections 2024-04-24T23:50:47.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.035+0000 7feb6ad30640 1 --2- 172.21.15.16:0/3813836442 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb6406bf40 0x7feb641413c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.035+0000 7feb6ad30640 1 --2- 172.21.15.16:0/3813836442 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb64071f40 0x7feb6406ba00 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.035+0000 7feb6ad30640 1 --2- 172.21.15.16:0/3813836442 >> v2:172.21.15.156:3300/0 conn(0x7feb6413eba0 0x7feb64071a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.035+0000 7feb6ad30640 1 -- 172.21.15.16:0/3813836442 >> 172.21.15.16:0/3813836442 conn(0x7feb640d0ac0 msgr2=0x7feb640d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.035+0000 7feb6ad30640 1 -- 172.21.15.16:0/3813836442 shutdown_connections 2024-04-24T23:50:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.035+0000 7feb6ad30640 1 -- 172.21.15.16:0/3813836442 wait complete. 2024-04-24T23:50:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.036+0000 7feb6ad30640 1 Processor -- start 2024-04-24T23:50:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.036+0000 7feb6ad30640 1 -- start start 2024-04-24T23:50:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb6ad30640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6406bf40 0x7feb64139c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb6ad30640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb64071f40 0x7feb640b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb6ad30640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb6413eba0 0x7feb6413f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb6ad30640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7feb6406c3d0 con 0x7feb6406bf40 2024-04-24T23:50:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb6ad30640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7feb6413ef80 con 0x7feb6413eba0 2024-04-24T23:50:47.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb6ad30640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7feb6406c550 con 0x7feb64071f40 2024-04-24T23:50:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb68acc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6406bf40 0x7feb64139c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb68acc640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6406bf40 0x7feb64139c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48872/0 (socket says 172.21.15.16:48872) 2024-04-24T23:50:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb68acc640 1 -- 172.21.15.16:0/3246408107 learned_addr learned my addr 172.21.15.16:0/3246408107 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb63fff640 1 --2- 172.21.15.16:0/3246408107 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb64071f40 0x7feb640b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb68acc640 1 -- 172.21.15.16:0/3246408107 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb64071f40 msgr2=0x7feb640b3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb68acc640 1 --2- 172.21.15.16:0/3246408107 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb64071f40 0x7feb640b3ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb692cd640 1 --2- 172.21.15.16:0/3246408107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb6413eba0 0x7feb6413f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb68acc640 1 -- 172.21.15.16:0/3246408107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb6413eba0 msgr2=0x7feb6413f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb68acc640 1 --2- 172.21.15.16:0/3246408107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb6413eba0 0x7feb6413f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb68acc640 1 -- 172.21.15.16:0/3246408107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb640b2a40 con 0x7feb6406bf40 2024-04-24T23:50:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.037+0000 7feb68acc640 1 --2- 172.21.15.16:0/3246408107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6406bf40 0x7feb64139c50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7feb5800b4a0 tx=0x7feb5800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:47.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.038+0000 7feb617fa640 1 -- 172.21.15.16:0/3246408107 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb58014070 con 0x7feb6406bf40 2024-04-24T23:50:47.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.038+0000 7feb617fa640 1 -- 172.21.15.16:0/3246408107 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb58010610 con 0x7feb6406bf40 2024-04-24T23:50:47.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.038+0000 7feb6ad30640 1 -- 172.21.15.16:0/3246408107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb640b2d20 con 0x7feb6406bf40 2024-04-24T23:50:47.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.038+0000 7feb617fa640 1 -- 172.21.15.16:0/3246408107 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb58010a00 con 0x7feb6406bf40 2024-04-24T23:50:47.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.038+0000 7feb6ad30640 1 -- 172.21.15.16:0/3246408107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb640b32e0 con 0x7feb6406bf40 2024-04-24T23:50:47.041 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.039+0000 7feb6ad30640 1 -- 172.21.15.16:0/3246408107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb6406f8f0 con 0x7feb6406bf40 2024-04-24T23:50:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.039+0000 7feb617fa640 1 -- 172.21.15.16:0/3246408107 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7feb58017070 con 0x7feb6406bf40 2024-04-24T23:50:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.040+0000 7feb617fa640 1 --2- 172.21.15.16:0/3246408107 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7feb4806be30 0x7feb4806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.040+0000 7feb617fa640 1 -- 172.21.15.16:0/3246408107 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7feb5808a860 con 0x7feb6406bf40 2024-04-24T23:50:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.040+0000 7feb63fff640 1 --2- 172.21.15.16:0/3246408107 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7feb4806be30 0x7feb4806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:47.042 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.040+0000 7feb63fff640 1 --2- 172.21.15.16:0/3246408107 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7feb4806be30 0x7feb4806e2f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7feb50034000 tx=0x7feb5002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:47.043 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.044+0000 7feb617fa640 1 -- 172.21.15.16:0/3246408107 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7feb58059290 con 0x7feb6406bf40 2024-04-24T23:50:47.302 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.302+0000 7feb6ad30640 1 -- 172.21.15.16:0/3246408107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_recovery_fs.TestFSRecovery.test_recover_fs_after_fsmap_removal"]} v 0) -- 0x7feb6404f5e0 con 0x7feb6406bf40 2024-04-24T23:50:47.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.580+0000 7feb617fa640 1 -- 172.21.15.16:0/3246408107 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_recovery_fs.TestFSRecovery.test_recover_fs_after_fsmap_removal"]}]=0 v130) ==== 159+0+0 (secure 0 0 0) 0x7feb58058b90 con 0x7feb6406bf40 2024-04-24T23:50:47.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.582+0000 7feb6ad30640 1 -- 172.21.15.16:0/3246408107 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7feb4806be30 msgr2=0x7feb4806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:47.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.582+0000 7feb6ad30640 1 --2- 172.21.15.16:0/3246408107 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7feb4806be30 0x7feb4806e2f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7feb50034000 tx=0x7feb5002e000 comp rx=0 tx=0).stop 2024-04-24T23:50:47.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.582+0000 7feb6ad30640 1 -- 172.21.15.16:0/3246408107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6406bf40 msgr2=0x7feb64139c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:47.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.582+0000 7feb6ad30640 1 --2- 172.21.15.16:0/3246408107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6406bf40 0x7feb64139c50 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7feb5800b4a0 tx=0x7feb5800b960 comp rx=0 tx=0).stop 2024-04-24T23:50:47.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.582+0000 7feb6ad30640 1 -- 172.21.15.16:0/3246408107 shutdown_connections 2024-04-24T23:50:47.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.582+0000 7feb6ad30640 1 --2- 172.21.15.16:0/3246408107 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7feb4806be30 0x7feb4806e2f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:47.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.582+0000 7feb6ad30640 1 --2- 172.21.15.16:0/3246408107 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb6413eba0 0x7feb6413f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:47.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.582+0000 7feb6ad30640 1 --2- 172.21.15.16:0/3246408107 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7feb64071f40 0x7feb640b3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:47.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.582+0000 7feb6ad30640 1 --2- 172.21.15.16:0/3246408107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb6406bf40 0x7feb64139c50 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:47.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.582+0000 7feb6ad30640 1 -- 172.21.15.16:0/3246408107 >> 172.21.15.16:0/3246408107 conn(0x7feb640d0ac0 msgr2=0x7feb6413dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:47.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.583+0000 7feb6ad30640 1 -- 172.21.15.16:0/3246408107 shutdown_connections 2024-04-24T23:50:47.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:47.583+0000 7feb6ad30640 1 -- 172.21.15.16:0/3246408107 wait complete. 2024-04-24T23:50:47.594 INFO:tasks.cephfs_test_runner:test_recover_fs_after_fsmap_removal (tasks.cephfs.test_recovery_fs.TestFSRecovery) ... ok 2024-04-24T23:50:47.594 INFO:tasks.cephfs_test_runner: 2024-04-24T23:50:47.594 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-04-24T23:50:47.594 INFO:tasks.cephfs_test_runner:Ran 1 test in 62.062s 2024-04-24T23:50:47.594 INFO:tasks.cephfs_test_runner: 2024-04-24T23:50:47.597 INFO:tasks.cephfs_test_runner:OK 2024-04-24T23:50:47.597 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-04-24T23:50:47.606 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-04-24T23:50:47.625 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-04-24T23:50:47.626 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:50:47.626 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:50:47.642 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056882,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19453 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T23:50:47.642 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-04-24T23:50:47.642 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:50:47.698 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056882,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19453 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249464k,nr_inodes=812366,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498932k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-04-24T23:50:47.699 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:50:47.699 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:50:47.718 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056108,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1214 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T23:50:47.718 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-04-24T23:50:47.718 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T23:50:47.777 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056108,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1214 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysusers.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-04-24T23:50:47.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:47.777 DEBUG:teuthology.orchestra.run.smithi016:> ip netns list 2024-04-24T23:50:47.793 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-04-24T23:50:47.793 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T23:50:47.793 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:47.794 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-04-24T23:50:47.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:47.859 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T23:50:47.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:47.926 DEBUG:teuthology.orchestra.run.smithi016:> sudo ip link delete ceph-brx 2024-04-24T23:50:48.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:48.031 DEBUG:teuthology.orchestra.run.smithi156:> ip netns list 2024-04-24T23:50:48.050 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-04-24T23:50:48.050 INFO:teuthology.orchestra.run.smithi156.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-04-24T23:50:48.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:48.050 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-04-24T23:50:48.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:48.120 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-04-24T23:50:48.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T23:50:48.188 DEBUG:teuthology.orchestra.run.smithi156:> sudo ip link delete ceph-brx 2024-04-24T23:50:48.286 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-04-24T23:50:48.299 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-04-24T23:50:48.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:50:48.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.405+0000 7fb7c831b640 1 Processor -- start 2024-04-24T23:50:48.405 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.405+0000 7fb7c831b640 1 -- start start 2024-04-24T23:50:48.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.406+0000 7fb7c831b640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb7c01457b0 0x7fb7c00ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:48.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.406+0000 7fb7c831b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7c015f990 0x7fb7c00cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:48.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.406+0000 7fb7c831b640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7c015fe90 con 0x7fb7c0144b20 2024-04-24T23:50:48.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.406+0000 7fb7c831b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb7c006ea50 con 0x7fb7c015f990 2024-04-24T23:50:48.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.406+0000 7fb7c831b640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fb7c0160170 con 0x7fb7c01457b0 2024-04-24T23:50:48.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.406+0000 7fb7c68b8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7c015f990 0x7fb7c00cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:48.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.406+0000 7fb7c68b8640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7c015f990 0x7fb7c00cebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48888/0 (socket says 172.21.15.16:48888) 2024-04-24T23:50:48.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.406+0000 7fb7c68b8640 1 -- 172.21.15.16:0/2992652904 learned_addr learned my addr 172.21.15.16:0/2992652904 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:48.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.406+0000 7fb7c58b6640 1 --2- 172.21.15.16:0/2992652904 >> v2:172.21.15.156:3300/0 conn(0x7fb7c01457b0 0x7fb7c00ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:48.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.406+0000 7fb7c68b8640 1 -- 172.21.15.16:0/2992652904 >> v2:172.21.15.156:3300/0 conn(0x7fb7c01457b0 msgr2=0x7fb7c00ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:48.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.406+0000 7fb7c68b8640 1 --2- 172.21.15.16:0/2992652904 >> v2:172.21.15.156:3300/0 conn(0x7fb7c01457b0 0x7fb7c00ce680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.406+0000 7fb7c68b8640 1 -- 172.21.15.16:0/2992652904 >> v1:172.21.15.156:6789/0 conn(0x7fb7c0144b20 legacy=0x7fb7c0144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:48.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.407+0000 7fb7c68b8640 1 -- 172.21.15.16:0/2992652904 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c0160450 con 0x7fb7c015f990 2024-04-24T23:50:48.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.407+0000 7fb7c68b8640 1 --2- 172.21.15.16:0/2992652904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7c015f990 0x7fb7c00cebc0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00b4ca0 tx=0x7fb7b802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b572dd69ba8405f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:48.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.407+0000 7fb7c50b5640 1 -- 172.21.15.16:0/2992652904 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b8036070 con 0x7fb7c015f990 2024-04-24T23:50:48.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.407+0000 7fb7c50b5640 1 -- 172.21.15.16:0/2992652904 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7b8031600 con 0x7fb7c015f990 2024-04-24T23:50:48.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.407+0000 7fb7c50b5640 1 -- 172.21.15.16:0/2992652904 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7b80319d0 con 0x7fb7c015f990 2024-04-24T23:50:48.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.408+0000 7fb7c831b640 1 -- 172.21.15.16:0/2992652904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7c015f990 msgr2=0x7fb7c00cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:48.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.408+0000 7fb7c831b640 1 --2- 172.21.15.16:0/2992652904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7c015f990 0x7fb7c00cebc0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fb7c00b4ca0 tx=0x7fb7b802a6a0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.408+0000 7fb7c831b640 1 -- 172.21.15.16:0/2992652904 shutdown_connections 2024-04-24T23:50:48.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.408+0000 7fb7c831b640 1 --2- 172.21.15.16:0/2992652904 >> v2:172.21.15.156:3300/0 conn(0x7fb7c01457b0 0x7fb7c00ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.408+0000 7fb7c831b640 1 --2- 172.21.15.16:0/2992652904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7c015f990 0x7fb7c00cebc0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.408+0000 7fb7c831b640 1 -- 172.21.15.16:0/2992652904 >> 172.21.15.16:0/2992652904 conn(0x7fb7c00b2000 msgr2=0x7fb7c00cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:48.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.408+0000 7fb7c831b640 1 -- 172.21.15.16:0/2992652904 shutdown_connections 2024-04-24T23:50:48.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.408+0000 7fb7c831b640 1 -- 172.21.15.16:0/2992652904 wait complete. 2024-04-24T23:50:48.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.409+0000 7fb7c831b640 1 Processor -- start 2024-04-24T23:50:48.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.409+0000 7fb7c831b640 1 -- start start 2024-04-24T23:50:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c831b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7c0144b20 0x7fb7c015cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c831b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb7c01457b0 0x7fb7c0157140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c831b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7c015f990 0x7fb7c0142370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c831b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c00b3390 con 0x7fb7c0144b20 2024-04-24T23:50:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c831b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb7c006d4b0 con 0x7fb7c015f990 2024-04-24T23:50:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c831b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb7c006da40 con 0x7fb7c01457b0 2024-04-24T23:50:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c60b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7c0144b20 0x7fb7c015cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:48.413 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c60b7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7c0144b20 0x7fb7c015cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48896/0 (socket says 172.21.15.16:48896) 2024-04-24T23:50:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c60b7640 1 -- 172.21.15.16:0/882061749 learned_addr learned my addr 172.21.15.16:0/882061749 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c68b8640 1 --2- 172.21.15.16:0/882061749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7c015f990 0x7fb7c0142370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c58b6640 1 --2- 172.21.15.16:0/882061749 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb7c01457b0 0x7fb7c0157140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c60b7640 1 -- 172.21.15.16:0/882061749 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb7c01457b0 msgr2=0x7fb7c0157140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c60b7640 1 --2- 172.21.15.16:0/882061749 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb7c01457b0 0x7fb7c0157140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c60b7640 1 -- 172.21.15.16:0/882061749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7c015f990 msgr2=0x7fb7c0142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c60b7640 1 --2- 172.21.15.16:0/882061749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7c015f990 0x7fb7c0142370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.410+0000 7fb7c60b7640 1 -- 172.21.15.16:0/882061749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7c013bfc0 con 0x7fb7c0144b20 2024-04-24T23:50:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.411+0000 7fb7c60b7640 1 --2- 172.21.15.16:0/882061749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7c0144b20 0x7fb7c015cdd0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c430 tx=0x7fb7bc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.411+0000 7fb7b6ffd640 1 -- 172.21.15.16:0/882061749 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc0052d0 con 0x7fb7c0144b20 2024-04-24T23:50:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.411+0000 7fb7b6ffd640 1 -- 172.21.15.16:0/882061749 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7bc005490 con 0x7fb7c0144b20 2024-04-24T23:50:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.411+0000 7fb7c831b640 1 -- 172.21.15.16:0/882061749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7c013ba00 con 0x7fb7c0144b20 2024-04-24T23:50:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.411+0000 7fb7b6ffd640 1 -- 172.21.15.16:0/882061749 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7bc007260 con 0x7fb7c0144b20 2024-04-24T23:50:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.411+0000 7fb7c831b640 1 -- 172.21.15.16:0/882061749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7c015c540 con 0x7fb7c0144b20 2024-04-24T23:50:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.412+0000 7fb7c831b640 1 -- 172.21.15.16:0/882061749 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7c015c740 con 0x7fb7c0144b20 2024-04-24T23:50:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.412+0000 7fb7b6ffd640 1 -- 172.21.15.16:0/882061749 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb7bc007400 con 0x7fb7c0144b20 2024-04-24T23:50:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.413+0000 7fb7b6ffd640 1 --2- 172.21.15.16:0/882061749 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb7a406be60 0x7fb7a406e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.413+0000 7fb7b6ffd640 1 -- 172.21.15.16:0/882061749 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7fb7bc08a1d0 con 0x7fb7c0144b20 2024-04-24T23:50:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.413+0000 7fb7c58b6640 1 --2- 172.21.15.16:0/882061749 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb7a406be60 0x7fb7a406e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:48.415 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.413+0000 7fb7c58b6640 1 --2- 172.21.15.16:0/882061749 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb7a406be60 0x7fb7a406e320 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0070d80 tx=0x7fb7b0008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:48.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.417+0000 7fb7b6ffd640 1 -- 172.21.15.16:0/882061749 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb7bc058d30 con 0x7fb7c0144b20 2024-04-24T23:50:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.598+0000 7fb7c831b640 1 -- 172.21.15.16:0/882061749 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb7c00731f0 con 0x7fb7a406be60 2024-04-24T23:50:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.599+0000 7fb7b6ffd640 1 -- 172.21.15.16:0/882061749 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13382 (secure 0 0 0) 0x7fb7c00731f0 con 0x7fb7a406be60 2024-04-24T23:50:48.599 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:48.599 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-24T23:50:48.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.601+0000 7fb7c831b640 1 -- 172.21.15.16:0/882061749 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb7a406be60 msgr2=0x7fb7a406e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:48.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.601+0000 7fb7c831b640 1 --2- 172.21.15.16:0/882061749 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb7a406be60 0x7fb7a406e320 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb7c0070d80 tx=0x7fb7b0008040 comp rx=0 tx=0).stop 2024-04-24T23:50:48.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.601+0000 7fb7c831b640 1 -- 172.21.15.16:0/882061749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7c0144b20 msgr2=0x7fb7c015cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:48.601 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.601+0000 7fb7c831b640 1 --2- 172.21.15.16:0/882061749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7c0144b20 0x7fb7c015cdd0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb7bc00c430 tx=0x7fb7bc00c8f0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.601+0000 7fb7c831b640 1 -- 172.21.15.16:0/882061749 shutdown_connections 2024-04-24T23:50:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.601+0000 7fb7c831b640 1 --2- 172.21.15.16:0/882061749 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb7a406be60 0x7fb7a406e320 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.601+0000 7fb7c831b640 1 --2- 172.21.15.16:0/882061749 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb7c015f990 0x7fb7c0142370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.601+0000 7fb7c831b640 1 --2- 172.21.15.16:0/882061749 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb7c01457b0 0x7fb7c0157140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.601+0000 7fb7c831b640 1 --2- 172.21.15.16:0/882061749 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7c0144b20 0x7fb7c015cdd0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.601+0000 7fb7c831b640 1 -- 172.21.15.16:0/882061749 >> 172.21.15.16:0/882061749 conn(0x7fb7c00b2000 msgr2=0x7fb7c006ddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.601+0000 7fb7c831b640 1 -- 172.21.15.16:0/882061749 shutdown_connections 2024-04-24T23:50:48.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.601+0000 7fb7c831b640 1 -- 172.21.15.16:0/882061749 wait complete. 2024-04-24T23:50:48.611 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-04-24T23:50:47.146790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254592,"kb_used_data":5144,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720128,"statfs":{"total":773094113280,"available":772833411072,"internally_reserved":0,"allocated":5267456,"data_stored":2968538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7208,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-79,"num_read_kb":-87,"num_write":-55,"num_write_kb":-24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001541"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":214,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T23:50:37.293613+0000","last_change":"2024-04-24T23:48:45.082459+0000","last_active":"2024-04-24T23:50:37.293613+0000","last_peered":"2024-04-24T23:50:37.293613+0000","last_clean":"2024-04-24T23:50:37.293613+0000","last_became_active":"2024-04-24T23:48:45.081495+0000","last_became_peered":"2024-04-24T23:48:45.081495+0000","last_unstale":"2024-04-24T23:50:37.293613+0000","last_undegraded":"2024-04-24T23:50:37.293613+0000","last_fullsized":"2024-04-24T23:50:37.293613+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_clean_scrub_stamp":"2024-04-24T23:48:43.440993+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:53:20.832511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1118208,"data_stored":819827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":221820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":221351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":507904,"data_stored":221842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":520192,"data_stored":229422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":221308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":221328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":811640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:50:48.612 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:50:48.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.712+0000 7f9f17058640 1 Processor -- start 2024-04-24T23:50:48.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.713+0000 7f9f17058640 1 -- start start 2024-04-24T23:50:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.713+0000 7f9f17058640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f10071a00 0x7f9f10071de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.713+0000 7f9f17058640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f100ce7c0 0x7f9f100cec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.713+0000 7f9f17058640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f10072320 con 0x7f9f10144740 2024-04-24T23:50:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.713+0000 7f9f17058640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f9f1015c1f0 con 0x7f9f100ce7c0 2024-04-24T23:50:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.713+0000 7f9f17058640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f9f100cf3a0 con 0x7f9f10071a00 2024-04-24T23:50:48.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.713+0000 7f9f155f5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f100ce7c0 0x7f9f100cec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.713+0000 7f9f155f5640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f100ce7c0 0x7f9f100cec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48904/0 (socket says 172.21.15.16:48904) 2024-04-24T23:50:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.714+0000 7f9f155f5640 1 -- 172.21.15.16:0/9331212 learned_addr learned my addr 172.21.15.16:0/9331212 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.714+0000 7f9f14df4640 1 --2- 172.21.15.16:0/9331212 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f10071a00 0x7f9f10071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.714+0000 7f9f155f5640 1 -- 172.21.15.16:0/9331212 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f10071a00 msgr2=0x7f9f10071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.714+0000 7f9f155f5640 1 --2- 172.21.15.16:0/9331212 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f10071a00 0x7f9f10071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.714+0000 7f9f155f5640 1 -- 172.21.15.16:0/9331212 >> v1:172.21.15.156:6789/0 conn(0x7f9f10144740 legacy=0x7f9f1015c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.714+0000 7f9f155f5640 1 -- 172.21.15.16:0/9331212 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f100cc8d0 con 0x7f9f100ce7c0 2024-04-24T23:50:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.714+0000 7f9f14df4640 1 --2- 172.21.15.16:0/9331212 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f10071a00 0x7f9f10071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.715+0000 7f9f155f5640 1 --2- 172.21.15.16:0/9331212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f100ce7c0 0x7f9f100cec70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9f00009870 tx=0x7f9f0002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=605247d4728bea53 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.715+0000 7f9f0f7fe640 1 -- 172.21.15.16:0/9331212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f00037070 con 0x7f9f100ce7c0 2024-04-24T23:50:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.715+0000 7f9f0f7fe640 1 -- 172.21.15.16:0/9331212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f00032440 con 0x7f9f100ce7c0 2024-04-24T23:50:48.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.715+0000 7f9f0f7fe640 1 -- 172.21.15.16:0/9331212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f00032810 con 0x7f9f100ce7c0 2024-04-24T23:50:48.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.715+0000 7f9f17058640 1 -- 172.21.15.16:0/9331212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f100ce7c0 msgr2=0x7f9f100cec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:48.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.715+0000 7f9f17058640 1 --2- 172.21.15.16:0/9331212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f100ce7c0 0x7f9f100cec70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f9f00009870 tx=0x7f9f0002a210 comp rx=0 tx=0).stop 2024-04-24T23:50:48.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.715+0000 7f9f17058640 1 -- 172.21.15.16:0/9331212 shutdown_connections 2024-04-24T23:50:48.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.715+0000 7f9f17058640 1 --2- 172.21.15.16:0/9331212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f100ce7c0 0x7f9f100cec70 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.715+0000 7f9f17058640 1 --2- 172.21.15.16:0/9331212 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f10071a00 0x7f9f10071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.718 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.715+0000 7f9f17058640 1 -- 172.21.15.16:0/9331212 >> 172.21.15.16:0/9331212 conn(0x7f9f100b2000 msgr2=0x7f9f100cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:48.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.715+0000 7f9f17058640 1 -- 172.21.15.16:0/9331212 shutdown_connections 2024-04-24T23:50:48.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.715+0000 7f9f17058640 1 -- 172.21.15.16:0/9331212 wait complete. 2024-04-24T23:50:48.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.716+0000 7f9f17058640 1 Processor -- start 2024-04-24T23:50:48.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.717+0000 7f9f17058640 1 -- start start 2024-04-24T23:50:48.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.717+0000 7f9f17058640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f10071a00 0x7f9f101f5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:48.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.717+0000 7f9f17058640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f100ce7c0 0x7f9f101f1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:48.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.717+0000 7f9f17058640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f101f1aa0 0x7f9f1006d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:48.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.717+0000 7f9f17058640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9f100cd170 con 0x7f9f10071a00 2024-04-24T23:50:48.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.717+0000 7f9f17058640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9f100cce90 con 0x7f9f100ce7c0 2024-04-24T23:50:48.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.717+0000 7f9f17058640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f9f1015cab0 con 0x7f9f101f1aa0 2024-04-24T23:50:48.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.717+0000 7f9f14df4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f10071a00 0x7f9f101f5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:48.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.717+0000 7f9f14df4640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f10071a00 0x7f9f101f5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48906/0 (socket says 172.21.15.16:48906) 2024-04-24T23:50:48.719 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.717+0000 7f9f14df4640 1 -- 172.21.15.16:0/805544953 learned_addr learned my addr 172.21.15.16:0/805544953 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:48.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.718+0000 7f9f155f5640 1 --2- 172.21.15.16:0/805544953 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f101f1aa0 0x7f9f1006d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:48.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.718+0000 7f9f0ffff640 1 --2- 172.21.15.16:0/805544953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f100ce7c0 0x7f9f101f1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:48.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.718+0000 7f9f14df4640 1 -- 172.21.15.16:0/805544953 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f101f1aa0 msgr2=0x7f9f1006d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:48.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.718+0000 7f9f14df4640 1 --2- 172.21.15.16:0/805544953 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f101f1aa0 0x7f9f1006d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.718+0000 7f9f14df4640 1 -- 172.21.15.16:0/805544953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f100ce7c0 msgr2=0x7f9f101f1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:48.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.718+0000 7f9f14df4640 1 --2- 172.21.15.16:0/805544953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f100ce7c0 0x7f9f101f1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.718+0000 7f9f14df4640 1 -- 172.21.15.16:0/805544953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f101f6030 con 0x7f9f10071a00 2024-04-24T23:50:48.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.718+0000 7f9f14df4640 1 --2- 172.21.15.16:0/805544953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f10071a00 0x7f9f101f5af0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800cf60 tx=0x7f9f0800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:48.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.718+0000 7f9f0d7fa640 1 -- 172.21.15.16:0/805544953 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0801a070 con 0x7f9f10071a00 2024-04-24T23:50:48.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.718+0000 7f9f0d7fa640 1 -- 172.21.15.16:0/805544953 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f0800d840 con 0x7f9f10071a00 2024-04-24T23:50:48.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.718+0000 7f9f17058640 1 -- 172.21.15.16:0/805544953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f100adb30 con 0x7f9f10071a00 2024-04-24T23:50:48.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.718+0000 7f9f0d7fa640 1 -- 172.21.15.16:0/805544953 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0800dc10 con 0x7f9f10071a00 2024-04-24T23:50:48.720 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.718+0000 7f9f17058640 1 -- 172.21.15.16:0/805544953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f101f2270 con 0x7f9f10071a00 2024-04-24T23:50:48.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.719+0000 7f9f17058640 1 -- 172.21.15.16:0/805544953 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f100731f0 con 0x7f9f10071a00 2024-04-24T23:50:48.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.720+0000 7f9f0d7fa640 1 -- 172.21.15.16:0/805544953 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9f0801d070 con 0x7f9f10071a00 2024-04-24T23:50:48.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.720+0000 7f9f0d7fa640 1 --2- 172.21.15.16:0/805544953 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9efc06be30 0x7f9efc06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:48.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.720+0000 7f9f0d7fa640 1 -- 172.21.15.16:0/805544953 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f9f0808b520 con 0x7f9f10071a00 2024-04-24T23:50:48.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.720+0000 7f9f0ffff640 1 --2- 172.21.15.16:0/805544953 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9efc06be30 0x7f9efc06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:48.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.721+0000 7f9f0ffff640 1 --2- 172.21.15.16:0/805544953 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9efc06be30 0x7f9efc06e2f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9f04006fd0 tx=0x7f9f04008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:48.724 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.724+0000 7f9f0d7fa640 1 -- 172.21.15.16:0/805544953 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9f101f6030 con 0x7f9f10071a00 2024-04-24T23:50:48.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.904+0000 7f9f17058640 1 -- 172.21.15.16:0/805544953 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9f101e3470 con 0x7f9efc06be30 2024-04-24T23:50:48.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.905+0000 7f9f0d7fa640 1 -- 172.21.15.16:0/805544953 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13382 (secure 0 0 0) 0x7f9f101e3470 con 0x7f9efc06be30 2024-04-24T23:50:48.904 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:48.905 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-24T23:50:48.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.907+0000 7f9f17058640 1 -- 172.21.15.16:0/805544953 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9efc06be30 msgr2=0x7f9efc06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:48.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.907+0000 7f9f17058640 1 --2- 172.21.15.16:0/805544953 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9efc06be30 0x7f9efc06e2f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9f04006fd0 tx=0x7f9f04008040 comp rx=0 tx=0).stop 2024-04-24T23:50:48.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.907+0000 7f9f17058640 1 -- 172.21.15.16:0/805544953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f10071a00 msgr2=0x7f9f101f5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:48.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.907+0000 7f9f17058640 1 --2- 172.21.15.16:0/805544953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f10071a00 0x7f9f101f5af0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f9f0800cf60 tx=0x7f9f0800cf90 comp rx=0 tx=0).stop 2024-04-24T23:50:48.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.907+0000 7f9f17058640 1 -- 172.21.15.16:0/805544953 shutdown_connections 2024-04-24T23:50:48.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.907+0000 7f9f17058640 1 --2- 172.21.15.16:0/805544953 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9efc06be30 0x7f9efc06e2f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.907+0000 7f9f17058640 1 --2- 172.21.15.16:0/805544953 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9f101f1aa0 0x7f9f1006d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.907+0000 7f9f17058640 1 --2- 172.21.15.16:0/805544953 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9f100ce7c0 0x7f9f101f1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.907+0000 7f9f17058640 1 --2- 172.21.15.16:0/805544953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f10071a00 0x7f9f101f5af0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:48.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.907+0000 7f9f17058640 1 -- 172.21.15.16:0/805544953 >> 172.21.15.16:0/805544953 conn(0x7f9f100b2000 msgr2=0x7f9f10142d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:48.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.907+0000 7f9f17058640 1 -- 172.21.15.16:0/805544953 shutdown_connections 2024-04-24T23:50:48.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:48.907+0000 7f9f17058640 1 -- 172.21.15.16:0/805544953 wait complete. 2024-04-24T23:50:48.917 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-04-24T23:50:47.146790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":254592,"kb_used_data":5144,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754720128,"statfs":{"total":773094113280,"available":772833411072,"internally_reserved":0,"allocated":5267456,"data_stored":2968538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7208,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-79,"num_read_kb":-87,"num_write":-55,"num_write_kb":-24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001541"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":214,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-04-24T23:50:37.293613+0000","last_change":"2024-04-24T23:48:45.082459+0000","last_active":"2024-04-24T23:50:37.293613+0000","last_peered":"2024-04-24T23:50:37.293613+0000","last_clean":"2024-04-24T23:50:37.293613+0000","last_became_active":"2024-04-24T23:48:45.081495+0000","last_became_peered":"2024-04-24T23:48:45.081495+0000","last_unstale":"2024-04-24T23:50:37.293613+0000","last_undegraded":"2024-04-24T23:50:37.293613+0000","last_fullsized":"2024-04-24T23:50:37.293613+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_clean_scrub_stamp":"2024-04-24T23:48:43.440993+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:53:20.832511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32268,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339572,"statfs":{"total":96636764160,"available":96603721728,"internally_reserved":0,"allocated":1118208,"data_stored":819827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":221820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31680,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340160,"statfs":{"total":96636764160,"available":96604323840,"internally_reserved":0,"allocated":512000,"data_stored":221351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":507904,"data_stored":221842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574876,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":520192,"data_stored":229422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31672,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340168,"statfs":{"total":96636764160,"available":96604332032,"internally_reserved":0,"allocated":503808,"data_stored":221308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574875,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31676,"kb_used_data":492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340164,"statfs":{"total":96636764160,"available":96604327936,"internally_reserved":0,"allocated":503808,"data_stored":221328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1097728,"data_stored":811640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:50:48.918 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-04-24T23:50:48.918 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:50:49.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.019+0000 7fe3a6163640 1 Processor -- start 2024-04-24T23:50:49.019 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.019+0000 7fe3a6163640 1 -- start start 2024-04-24T23:50:49.020 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe3a6163640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fe3a0144b20 0x7fe3a0144f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe3a6163640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3a01457b0 0x7fe3a00ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe3a6163640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3a015f990 0x7fe3a00cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe3a6163640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fe3a0145440 con 0x7fe3a015f990 2024-04-24T23:50:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe3a6163640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fe3a015fe90 con 0x7fe3a0144b20 2024-04-24T23:50:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe3a6163640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe3a0160170 con 0x7fe3a01457b0 2024-04-24T23:50:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe39ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3a015f990 0x7fe3a00cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe39ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe3a015f990 0x7fe3a00cebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48912/0 (socket says 172.21.15.16:48912) 2024-04-24T23:50:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe39ffff640 1 -- 172.21.15.16:0/1161915224 learned_addr learned my addr 172.21.15.16:0/1161915224 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe39f7fe640 1 --2- 172.21.15.16:0/1161915224 >> v2:172.21.15.156:3300/0 conn(0x7fe3a0144b20 0x7fe3a0144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe39effd640 1 --2- 172.21.15.16:0/1161915224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3a01457b0 0x7fe3a00ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe39ffff640 1 -- 172.21.15.16:0/1161915224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3a01457b0 msgr2=0x7fe3a00ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe39ffff640 1 --2- 172.21.15.16:0/1161915224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3a01457b0 0x7fe3a00ce680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe39ffff640 1 -- 172.21.15.16:0/1161915224 >> v2:172.21.15.156:3300/0 conn(0x7fe3a0144b20 msgr2=0x7fe3a0144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:49.021 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe39ffff640 1 --2- 172.21.15.16:0/1161915224 >> v2:172.21.15.156:3300/0 conn(0x7fe3a0144b20 0x7fe3a0144f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe39ffff640 1 -- 172.21.15.16:0/1161915224 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a0160450 con 0x7fe3a015f990 2024-04-24T23:50:49.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.020+0000 7fe39f7fe640 1 --2- 172.21.15.16:0/1161915224 >> v2:172.21.15.156:3300/0 conn(0x7fe3a0144b20 0x7fe3a0144f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:49.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.021+0000 7fe39ffff640 1 --2- 172.21.15.16:0/1161915224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a015f990 0x7fe3a00cebc0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0072ea0 tx=0x7fe39802a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee47219846c4447c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:49.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.021+0000 7fe39e7fc640 1 -- 172.21.15.16:0/1161915224 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398036070 con 0x7fe3a015f990 2024-04-24T23:50:49.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.021+0000 7fe39e7fc640 1 -- 172.21.15.16:0/1161915224 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe398031600 con 0x7fe3a015f990 2024-04-24T23:50:49.022 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.021+0000 7fe39e7fc640 1 -- 172.21.15.16:0/1161915224 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3980319d0 con 0x7fe3a015f990 2024-04-24T23:50:49.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.021+0000 7fe3a6163640 1 -- 172.21.15.16:0/1161915224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a015f990 msgr2=0x7fe3a00cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:49.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.022+0000 7fe3a6163640 1 --2- 172.21.15.16:0/1161915224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a015f990 0x7fe3a00cebc0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0072ea0 tx=0x7fe39802a610 comp rx=0 tx=0).stop 2024-04-24T23:50:49.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.022+0000 7fe3a6163640 1 -- 172.21.15.16:0/1161915224 shutdown_connections 2024-04-24T23:50:49.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.022+0000 7fe3a6163640 1 --2- 172.21.15.16:0/1161915224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a015f990 0x7fe3a00cebc0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.022+0000 7fe3a6163640 1 --2- 172.21.15.16:0/1161915224 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3a01457b0 0x7fe3a00ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.022+0000 7fe3a6163640 1 --2- 172.21.15.16:0/1161915224 >> v2:172.21.15.156:3300/0 conn(0x7fe3a0144b20 0x7fe3a0144f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.022+0000 7fe3a6163640 1 -- 172.21.15.16:0/1161915224 >> 172.21.15.16:0/1161915224 conn(0x7fe3a00b2000 msgr2=0x7fe3a00cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:49.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.022+0000 7fe3a6163640 1 -- 172.21.15.16:0/1161915224 shutdown_connections 2024-04-24T23:50:49.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.022+0000 7fe3a6163640 1 -- 172.21.15.16:0/1161915224 wait complete. 2024-04-24T23:50:49.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.023+0000 7fe3a6163640 1 Processor -- start 2024-04-24T23:50:49.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.023+0000 7fe3a6163640 1 -- start start 2024-04-24T23:50:49.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe3a6163640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3a0144b20 0x7fe3a013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:49.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe3a6163640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a01457b0 0x7fe3a0157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:49.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe3a6163640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3a015f990 0x7fe3a01406f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:49.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe3a6163640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a00b3390 con 0x7fe3a01457b0 2024-04-24T23:50:49.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe3a6163640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a006d4b0 con 0x7fe3a015f990 2024-04-24T23:50:49.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe3a6163640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fe3a0071a00 con 0x7fe3a0144b20 2024-04-24T23:50:49.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe39effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a01457b0 0x7fe3a0157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:49.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe39effd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a01457b0 0x7fe3a0157460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48914/0 (socket says 172.21.15.16:48914) 2024-04-24T23:50:49.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe39effd640 1 -- 172.21.15.16:0/2301345905 learned_addr learned my addr 172.21.15.16:0/2301345905 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:49.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe39f7fe640 1 --2- 172.21.15.16:0/2301345905 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3a0144b20 0x7fe3a013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:49.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe39ffff640 1 --2- 172.21.15.16:0/2301345905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3a015f990 0x7fe3a01406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:49.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe39effd640 1 -- 172.21.15.16:0/2301345905 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3a0144b20 msgr2=0x7fe3a013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:49.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe39effd640 1 --2- 172.21.15.16:0/2301345905 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3a0144b20 0x7fe3a013ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe39effd640 1 -- 172.21.15.16:0/2301345905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3a015f990 msgr2=0x7fe3a01406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:49.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe39effd640 1 --2- 172.21.15.16:0/2301345905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3a015f990 0x7fe3a01406f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe39effd640 1 -- 172.21.15.16:0/2301345905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a013d760 con 0x7fe3a01457b0 2024-04-24T23:50:49.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe39effd640 1 --2- 172.21.15.16:0/2301345905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a01457b0 0x7fe3a0157460 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c7f0 tx=0x7fe39400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:49.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe387fff640 1 -- 172.21.15.16:0/2301345905 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394015070 con 0x7fe3a01457b0 2024-04-24T23:50:49.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe387fff640 1 -- 172.21.15.16:0/2301345905 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe394004240 con 0x7fe3a01457b0 2024-04-24T23:50:49.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe387fff640 1 -- 172.21.15.16:0/2301345905 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe394007620 con 0x7fe3a01457b0 2024-04-24T23:50:49.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.024+0000 7fe3a6163640 1 -- 172.21.15.16:0/2301345905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3a013da40 con 0x7fe3a01457b0 2024-04-24T23:50:49.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.025+0000 7fe3a6163640 1 -- 172.21.15.16:0/2301345905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3a01579a0 con 0x7fe3a01457b0 2024-04-24T23:50:49.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.026+0000 7fe3a6163640 1 -- 172.21.15.16:0/2301345905 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3a00731f0 con 0x7fe3a01457b0 2024-04-24T23:50:49.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.026+0000 7fe387fff640 1 -- 172.21.15.16:0/2301345905 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe3940079d0 con 0x7fe3a01457b0 2024-04-24T23:50:49.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.027+0000 7fe387fff640 1 --2- 172.21.15.16:0/2301345905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe38c06be60 0x7fe38c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:49.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.027+0000 7fe387fff640 1 -- 172.21.15.16:0/2301345905 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7fe39408adb0 con 0x7fe3a01457b0 2024-04-24T23:50:49.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.027+0000 7fe39f7fe640 1 --2- 172.21.15.16:0/2301345905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe38c06be60 0x7fe38c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:49.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.027+0000 7fe39f7fe640 1 --2- 172.21.15.16:0/2301345905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe38c06be60 0x7fe38c06e320 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe390006fd0 tx=0x7fe390008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:49.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.031+0000 7fe387fff640 1 -- 172.21.15.16:0/2301345905 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fe394059910 con 0x7fe3a01457b0 2024-04-24T23:50:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.209+0000 7fe3a6163640 1 -- 172.21.15.16:0/2301345905 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe3a00b4bd0 con 0x7fe38c06be60 2024-04-24T23:50:49.210 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.210+0000 7fe387fff640 1 -- 172.21.15.16:0/2301345905 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13475 (secure 0 0 0) 0x7fe3a00b4bd0 con 0x7fe38c06be60 2024-04-24T23:50:49.210 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:49.210 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-24T23:50:49.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.212+0000 7fe3a6163640 1 -- 172.21.15.16:0/2301345905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe38c06be60 msgr2=0x7fe38c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:49.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.212+0000 7fe3a6163640 1 --2- 172.21.15.16:0/2301345905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe38c06be60 0x7fe38c06e320 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe390006fd0 tx=0x7fe390008040 comp rx=0 tx=0).stop 2024-04-24T23:50:49.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.212+0000 7fe3a6163640 1 -- 172.21.15.16:0/2301345905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a01457b0 msgr2=0x7fe3a0157460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:49.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.212+0000 7fe3a6163640 1 --2- 172.21.15.16:0/2301345905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a01457b0 0x7fe3a0157460 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c7f0 tx=0x7fe39400ccb0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.212 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.212+0000 7fe3a6163640 1 -- 172.21.15.16:0/2301345905 shutdown_connections 2024-04-24T23:50:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.212+0000 7fe3a6163640 1 --2- 172.21.15.16:0/2301345905 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fe38c06be60 0x7fe38c06e320 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.212+0000 7fe3a6163640 1 --2- 172.21.15.16:0/2301345905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe3a015f990 0x7fe3a01406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.212+0000 7fe3a6163640 1 --2- 172.21.15.16:0/2301345905 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe3a01457b0 0x7fe3a0157460 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.212+0000 7fe3a6163640 1 --2- 172.21.15.16:0/2301345905 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fe3a0144b20 0x7fe3a013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.212+0000 7fe3a6163640 1 -- 172.21.15.16:0/2301345905 >> 172.21.15.16:0/2301345905 conn(0x7fe3a00b2000 msgr2=0x7fe3a006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.212+0000 7fe3a6163640 1 -- 172.21.15.16:0/2301345905 shutdown_connections 2024-04-24T23:50:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.212+0000 7fe3a6163640 1 -- 172.21.15.16:0/2301345905 wait complete. 2024-04-24T23:50:49.223 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-04-24T23:50:49.147346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":115,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":254768,"kb_used_data":5316,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719952,"statfs":{"total":773094113280,"available":772833230848,"internally_reserved":0,"allocated":5443584,"data_stored":3060820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8949,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-80,"num_read_kb":-88,"num_write":-61,"num_write_kb":-28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001700"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":221,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T23:50:46.582781+0000","last_change":"2024-04-24T23:48:45.082459+0000","last_active":"2024-04-24T23:50:46.582781+0000","last_peered":"2024-04-24T23:50:46.582781+0000","last_clean":"2024-04-24T23:50:46.582781+0000","last_became_active":"2024-04-24T23:48:45.081495+0000","last_became_peered":"2024-04-24T23:48:45.081495+0000","last_unstale":"2024-04-24T23:50:46.582781+0000","last_undegraded":"2024-04-24T23:50:46.582781+0000","last_fullsized":"2024-04-24T23:50:46.582781+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_clean_scrub_stamp":"2024-04-24T23:48:43.440993+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:53:20.832511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1138688,"data_stored":830764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31684,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":516096,"data_stored":228408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31716,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574877,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31684,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340156,"statfs":{"total":96636764160,"available":96604319744,"internally_reserved":0,"allocated":516096,"data_stored":227896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31700,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340140,"statfs":{"total":96636764160,"available":96604303360,"internally_reserved":0,"allocated":528384,"data_stored":234336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32264,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339576,"statfs":{"total":96636764160,"available":96603725824,"internally_reserved":0,"allocated":1110016,"data_stored":818228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:50:49.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T23:50:49.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.325+0000 7f6286570640 1 Processor -- start 2024-04-24T23:50:49.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.326+0000 7f6286570640 1 -- start start 2024-04-24T23:50:49.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.326+0000 7f6286570640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f628015e020 0x7f62800cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:49.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.326+0000 7f6286570640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f62800b2bb0 0x7f62800cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:49.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.326+0000 7f6286570640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62800b3390 con 0x7f62800cfcf0 2024-04-24T23:50:49.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.326+0000 7f6286570640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f62800d0100 con 0x7f62800b2bb0 2024-04-24T23:50:49.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.326+0000 7f6286570640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f628006bfc0 con 0x7f628015e020 2024-04-24T23:50:49.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.326+0000 7f627f7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f62800b2bb0 0x7f62800cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:49.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.327+0000 7f627f7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f62800b2bb0 0x7f62800cf7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48922/0 (socket says 172.21.15.16:48922) 2024-04-24T23:50:49.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.327+0000 7f627f7fe640 1 -- 172.21.15.16:0/3048993192 learned_addr learned my addr 172.21.15.16:0/3048993192 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:49.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.327+0000 7f627ffff640 1 --2- 172.21.15.16:0/3048993192 >> v2:172.21.15.156:3300/0 conn(0x7f628015e020 0x7f62800cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:49.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.327+0000 7f627f7fe640 1 -- 172.21.15.16:0/3048993192 >> v2:172.21.15.156:3300/0 conn(0x7f628015e020 msgr2=0x7f62800cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:49.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.327+0000 7f627f7fe640 1 --2- 172.21.15.16:0/3048993192 >> v2:172.21.15.156:3300/0 conn(0x7f628015e020 0x7f62800cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.327+0000 7f627f7fe640 1 -- 172.21.15.16:0/3048993192 >> v1:172.21.15.156:6789/0 conn(0x7f62800cfcf0 legacy=0x7f6280161bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:49.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.327+0000 7f627f7fe640 1 -- 172.21.15.16:0/3048993192 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f628006c2a0 con 0x7f62800b2bb0 2024-04-24T23:50:49.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.327+0000 7f627f7fe640 1 --2- 172.21.15.16:0/3048993192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62800b2bb0 0x7f62800cf7b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f62800d3de0 tx=0x7f627002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f7c21da1f34fa9c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:49.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.328+0000 7f627effd640 1 -- 172.21.15.16:0/3048993192 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6270037070 con 0x7f62800b2bb0 2024-04-24T23:50:49.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.328+0000 7f627effd640 1 -- 172.21.15.16:0/3048993192 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6270032440 con 0x7f62800b2bb0 2024-04-24T23:50:49.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.328+0000 7f627effd640 1 -- 172.21.15.16:0/3048993192 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6270032810 con 0x7f62800b2bb0 2024-04-24T23:50:49.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.328+0000 7f6286570640 1 -- 172.21.15.16:0/3048993192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62800b2bb0 msgr2=0x7f62800cf7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:49.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.328+0000 7f6286570640 1 --2- 172.21.15.16:0/3048993192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62800b2bb0 0x7f62800cf7b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f62800d3de0 tx=0x7f627002a1e0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.328+0000 7f6286570640 1 -- 172.21.15.16:0/3048993192 shutdown_connections 2024-04-24T23:50:49.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.329+0000 7f6286570640 1 --2- 172.21.15.16:0/3048993192 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62800b2bb0 0x7f62800cf7b0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.329+0000 7f6286570640 1 --2- 172.21.15.16:0/3048993192 >> v2:172.21.15.156:3300/0 conn(0x7f628015e020 0x7f62800cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.329+0000 7f6286570640 1 -- 172.21.15.16:0/3048993192 >> 172.21.15.16:0/3048993192 conn(0x7f62800d3e10 msgr2=0x7f62800d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:49.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.329+0000 7f6286570640 1 -- 172.21.15.16:0/3048993192 shutdown_connections 2024-04-24T23:50:49.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.329+0000 7f6286570640 1 -- 172.21.15.16:0/3048993192 wait complete. 2024-04-24T23:50:49.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.330+0000 7f6286570640 1 Processor -- start 2024-04-24T23:50:49.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.331+0000 7f6286570640 1 -- start start 2024-04-24T23:50:49.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f6286570640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62800b2bb0 0x7f628014f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:49.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f6286570640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62800cfcf0 0x7f628015bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:49.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f6286570640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f628015e020 0x7f62800c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:49.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f6286570640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f62800d1a80 con 0x7f62800b2bb0 2024-04-24T23:50:49.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f6286570640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f628006c860 con 0x7f628015e020 2024-04-24T23:50:49.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f6286570640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f628015e400 con 0x7f62800cfcf0 2024-04-24T23:50:49.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f627ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62800b2bb0 0x7f628014f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:49.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f627ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62800b2bb0 0x7f628014f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48930/0 (socket says 172.21.15.16:48930) 2024-04-24T23:50:49.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f627ffff640 1 -- 172.21.15.16:0/2505641476 learned_addr learned my addr 172.21.15.16:0/2505641476 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:49.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f627f7fe640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62800cfcf0 0x7f628015bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:49.333 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f6284b0d640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f628015e020 0x7f62800c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:49.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f627ffff640 1 -- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62800cfcf0 msgr2=0x7f628015bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:49.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f627ffff640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62800cfcf0 0x7f628015bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f627ffff640 1 -- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f628015e020 msgr2=0x7f62800c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:49.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f627ffff640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f628015e020 0x7f62800c9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f627ffff640 1 -- 172.21.15.16:0/2505641476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62800c8990 con 0x7f62800b2bb0 2024-04-24T23:50:49.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.332+0000 7f627f7fe640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62800cfcf0 0x7f628015bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:49.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.333+0000 7f627ffff640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62800b2bb0 0x7f628014f1f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f626800e9b0 tx=0x7f626800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:49.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.333+0000 7f627cff9640 1 -- 172.21.15.16:0/2505641476 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6268017070 con 0x7f62800b2bb0 2024-04-24T23:50:49.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.333+0000 7f627cff9640 1 -- 172.21.15.16:0/2505641476 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f626800c9a0 con 0x7f62800b2bb0 2024-04-24T23:50:49.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.333+0000 7f627cff9640 1 -- 172.21.15.16:0/2505641476 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626800cd90 con 0x7f62800b2bb0 2024-04-24T23:50:49.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.333+0000 7f6286570640 1 -- 172.21.15.16:0/2505641476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62800c8c30 con 0x7f62800b2bb0 2024-04-24T23:50:49.334 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.333+0000 7f6286570640 1 -- 172.21.15.16:0/2505641476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62800c91f0 con 0x7f62800b2bb0 2024-04-24T23:50:49.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.334+0000 7f6286570640 1 -- 172.21.15.16:0/2505641476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f628006dfe0 con 0x7f62800b2bb0 2024-04-24T23:50:49.335 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.334+0000 7f627cff9640 1 -- 172.21.15.16:0/2505641476 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f626801a070 con 0x7f62800b2bb0 2024-04-24T23:50:49.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.335+0000 7f627cff9640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f626006be60 0x7f626006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:49.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.335+0000 7f627cff9640 1 -- 172.21.15.16:0/2505641476 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f626808acf0 con 0x7f62800b2bb0 2024-04-24T23:50:49.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.335+0000 7f627f7fe640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f626006be60 0x7f626006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:49.336 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.336+0000 7f627f7fe640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f626006be60 0x7f626006e320 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6270034000 tx=0x7f627002e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:49.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.339+0000 7f627cff9640 1 -- 172.21.15.16:0/2505641476 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f62680597b0 con 0x7f62800b2bb0 2024-04-24T23:50:49.522 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.523+0000 7f6286570640 1 -- 172.21.15.16:0/2505641476 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f62800ced00 con 0x7f62800b2bb0 2024-04-24T23:50:49.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.524+0000 7f627cff9640 1 -- 172.21.15.16:0/2505641476 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) ==== 74+0+13409 (secure 0 0 0) 0x7f62680590b0 con 0x7f62800b2bb0 2024-04-24T23:50:49.523 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:49.523 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":46,"fsid":"f8999c8c-ab57-4aba-8d53-08e19e6ee3d5","created":"2024-04-24T23:48:30.925296+0000","modified":"2024-04-24T23:50:46.573635+0000","last_up_change":"2024-04-24T23:48:41.437315+0000","last_in_change":"2024-04-24T23:48:33.060932+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T23:48:43.160738+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6f0cd63f-5346-49f1-a49c-909e5c367d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6825","nonce":2411432146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6827","nonce":2411432146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6831","nonce":2411432146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":2411432146},{"type":"v1","addr":"172.21.15.16:6829","nonce":2411432146}]},"public_addr":"172.21.15.16:6825/2411432146","cluster_addr":"172.21.15.16:6827/2411432146","heartbeat_back_addr":"172.21.15.16:6831/2411432146","heartbeat_front_addr":"172.21.15.16:6829/2411432146","state":["exists","up"]},{"osd":1,"uuid":"5624c321-08e4-4673-9bdb-9b657d33a93e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6801","nonce":2543873173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6803","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6805","nonce":2543873173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6811","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6813","nonce":2543873173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6807","nonce":2543873173},{"type":"v1","addr":"172.21.15.16:6809","nonce":2543873173}]},"public_addr":"172.21.15.16:6801/2543873173","cluster_addr":"172.21.15.16:6805/2543873173","heartbeat_back_addr":"172.21.15.16:6813/2543873173","heartbeat_front_addr":"172.21.15.16:6809/2543873173","state":["exists","up"]},{"osd":2,"uuid":"bf2cfb61-05a1-4c9f-a3c5-2cd568bb2162","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6804","nonce":4175732037}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6808","nonce":4175732037}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6815","nonce":4175732037}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":4175732037},{"type":"v1","addr":"172.21.15.16:6812","nonce":4175732037}]},"public_addr":"172.21.15.16:6804/4175732037","cluster_addr":"172.21.15.16:6808/4175732037","heartbeat_back_addr":"172.21.15.16:6815/4175732037","heartbeat_front_addr":"172.21.15.16:6812/4175732037","state":["exists","up"]},{"osd":3,"uuid":"72dbaef3-46da-45b1-9744-d8abb632614c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6817","nonce":3747244008}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6819","nonce":3747244008}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6823","nonce":3747244008}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":3747244008},{"type":"v1","addr":"172.21.15.16:6821","nonce":3747244008}]},"public_addr":"172.21.15.16:6817/3747244008","cluster_addr":"172.21.15.16:6819/3747244008","heartbeat_back_addr":"172.21.15.16:6823/3747244008","heartbeat_front_addr":"172.21.15.16:6821/3747244008","state":["exists","up"]},{"osd":4,"uuid":"75482b46-cd25-4196-8026-87fac8e73a41","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6809","nonce":542868407}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6811","nonce":542868407}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6815","nonce":542868407}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":542868407},{"type":"v1","addr":"172.21.15.156:6813","nonce":542868407}]},"public_addr":"172.21.15.156:6809/542868407","cluster_addr":"172.21.15.156:6811/542868407","heartbeat_back_addr":"172.21.15.156:6815/542868407","heartbeat_front_addr":"172.21.15.156:6813/542868407","state":["exists","up"]},{"osd":5,"uuid":"2f389c70-5f03-4d8b-b7d6-34f3fdc2b43d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6801","nonce":1918945722}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6803","nonce":1918945722}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6807","nonce":1918945722}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":1918945722},{"type":"v1","addr":"172.21.15.156:6805","nonce":1918945722}]},"public_addr":"172.21.15.156:6801/1918945722","cluster_addr":"172.21.15.156:6803/1918945722","heartbeat_back_addr":"172.21.15.156:6807/1918945722","heartbeat_front_addr":"172.21.15.156:6805/1918945722","state":["exists","up"]},{"osd":6,"uuid":"4f095fbc-1077-4a2d-bde6-98d4dd3b0931","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6817","nonce":667041555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6819","nonce":667041555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6823","nonce":667041555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":667041555},{"type":"v1","addr":"172.21.15.156:6821","nonce":667041555}]},"public_addr":"172.21.15.156:6817/667041555","cluster_addr":"172.21.15.156:6819/667041555","heartbeat_back_addr":"172.21.15.156:6823/667041555","heartbeat_front_addr":"172.21.15.156:6821/667041555","state":["exists","up"]},{"osd":7,"uuid":"3ba94c69-3667-4d5a-8f3e-9d30f3c784e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6825","nonce":2447858915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6827","nonce":2447858915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6831","nonce":2447858915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":2447858915},{"type":"v1","addr":"172.21.15.156:6829","nonce":2447858915}]},"public_addr":"172.21.15.156:6825/2447858915","cluster_addr":"172.21.15.156:6827/2447858915","heartbeat_back_addr":"172.21.15.156:6831/2447858915","heartbeat_front_addr":"172.21.15.156:6829/2447858915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.234433+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.154933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.156023+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.248337+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.274242+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.167589+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.293565+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T23:48:38.272000+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:6834/3643102469":"2024-04-25T23:50:43.566347+0000","172.21.15.16:6834/561619268":"2024-04-25T23:50:30.259891+0000","172.21.15.16:6832/3643102469":"2024-04-25T23:50:43.566347+0000","172.21.15.16:6832/561619268":"2024-04-25T23:50:30.259891+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T23:50:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.526+0000 7f6286570640 1 -- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f626006be60 msgr2=0x7f626006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.526+0000 7f6286570640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f626006be60 0x7f626006e320 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6270034000 tx=0x7f627002e000 comp rx=0 tx=0).stop 2024-04-24T23:50:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.526+0000 7f6286570640 1 -- 172.21.15.16:0/2505641476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62800b2bb0 msgr2=0x7f628014f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.526+0000 7f6286570640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62800b2bb0 0x7f628014f1f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f626800e9b0 tx=0x7f626800ee70 comp rx=0 tx=0).stop 2024-04-24T23:50:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.526+0000 7f6286570640 1 -- 172.21.15.16:0/2505641476 shutdown_connections 2024-04-24T23:50:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.526+0000 7f6286570640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f626006be60 0x7f626006e320 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.526+0000 7f6286570640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f628015e020 0x7f62800c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.526+0000 7f6286570640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f62800cfcf0 0x7f628015bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.526+0000 7f6286570640 1 --2- 172.21.15.16:0/2505641476 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f62800b2bb0 0x7f628014f1f0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.526+0000 7f6286570640 1 -- 172.21.15.16:0/2505641476 >> 172.21.15.16:0/2505641476 conn(0x7f62800d3e10 msgr2=0x7f62800b7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.526+0000 7f6286570640 1 -- 172.21.15.16:0/2505641476 shutdown_connections 2024-04-24T23:50:49.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:49.526+0000 7f6286570640 1 -- 172.21.15.16:0/2505641476 wait complete. 2024-04-24T23:50:50.537 INFO:tasks.ceph:Scrubbing osd.0 2024-04-24T23:50:50.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:50:50.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.644+0000 7f1106322640 1 Processor -- start 2024-04-24T23:50:50.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.644+0000 7f1106322640 1 -- start start 2024-04-24T23:50:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.645+0000 7f1106322640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1100145630 0x7f110006e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.645+0000 7f1106322640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f11000b54b0 0x7f1100071230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.645+0000 7f1106322640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f110006d7c0 con 0x7f110006e7c0 2024-04-24T23:50:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.645+0000 7f1106322640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f110006d560 con 0x7f1100145630 2024-04-24T23:50:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.645+0000 7f1106322640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f1100160660 con 0x7f11000b54b0 2024-04-24T23:50:50.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.645+0000 7f10fffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1100145630 0x7f110006e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.645+0000 7f10fffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1100145630 0x7f110006e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48946/0 (socket says 172.21.15.16:48946) 2024-04-24T23:50:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.645+0000 7f10fffff640 1 -- 172.21.15.16:0/2845956918 learned_addr learned my addr 172.21.15.16:0/2845956918 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.645+0000 7f11048bf640 1 --2- 172.21.15.16:0/2845956918 >> v2:172.21.15.156:3300/0 conn(0x7f11000b54b0 0x7f1100071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.645+0000 7f10fffff640 1 -- 172.21.15.16:0/2845956918 >> v2:172.21.15.156:3300/0 conn(0x7f11000b54b0 msgr2=0x7f1100071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.645+0000 7f10fffff640 1 --2- 172.21.15.16:0/2845956918 >> v2:172.21.15.156:3300/0 conn(0x7f11000b54b0 0x7f1100071230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.645+0000 7f10fffff640 1 -- 172.21.15.16:0/2845956918 >> v1:172.21.15.156:6789/0 conn(0x7f110006e7c0 legacy=0x7f11000b4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:50.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.646+0000 7f10fffff640 1 -- 172.21.15.16:0/2845956918 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f110006da80 con 0x7f1100145630 2024-04-24T23:50:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.646+0000 7f10fffff640 1 --2- 172.21.15.16:0/2845956918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1100145630 0x7f110006e280 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1100070a60 tx=0x7f10e802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9fb6413c6e3a3af server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.646+0000 7f10feffd640 1 -- 172.21.15.16:0/2845956918 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e8037070 con 0x7f1100145630 2024-04-24T23:50:50.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.646+0000 7f10feffd640 1 -- 172.21.15.16:0/2845956918 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10e8032440 con 0x7f1100145630 2024-04-24T23:50:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.646+0000 7f10feffd640 1 -- 172.21.15.16:0/2845956918 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e8032810 con 0x7f1100145630 2024-04-24T23:50:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.647+0000 7f1106322640 1 -- 172.21.15.16:0/2845956918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1100145630 msgr2=0x7f110006e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.647+0000 7f1106322640 1 --2- 172.21.15.16:0/2845956918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1100145630 0x7f110006e280 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f1100070a60 tx=0x7f10e802a230 comp rx=0 tx=0).stop 2024-04-24T23:50:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.647+0000 7f1106322640 1 -- 172.21.15.16:0/2845956918 shutdown_connections 2024-04-24T23:50:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.647+0000 7f1106322640 1 --2- 172.21.15.16:0/2845956918 >> v2:172.21.15.156:3300/0 conn(0x7f11000b54b0 0x7f1100071230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.647+0000 7f1106322640 1 --2- 172.21.15.16:0/2845956918 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1100145630 0x7f110006e280 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.647+0000 7f1106322640 1 -- 172.21.15.16:0/2845956918 >> 172.21.15.16:0/2845956918 conn(0x7f11000cfa20 msgr2=0x7f11000cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.647+0000 7f1106322640 1 -- 172.21.15.16:0/2845956918 shutdown_connections 2024-04-24T23:50:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.647+0000 7f1106322640 1 -- 172.21.15.16:0/2845956918 wait complete. 2024-04-24T23:50:50.648 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.648+0000 7f1106322640 1 Processor -- start 2024-04-24T23:50:50.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.648+0000 7f1106322640 1 -- start start 2024-04-24T23:50:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f1106322640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f110006e7c0 0x7f110013fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f1106322640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11000b54b0 0x7f11001e6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f1106322640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1100145630 0x7f11000c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f1106322640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f1100145b90 con 0x7f11000b54b0 2024-04-24T23:50:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f1106322640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1100145a10 con 0x7f1100145630 2024-04-24T23:50:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f1106322640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f1100071770 con 0x7f110006e7c0 2024-04-24T23:50:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f10ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11000b54b0 0x7f11001e6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f10ff7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11000b54b0 0x7f11001e6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48948/0 (socket says 172.21.15.16:48948) 2024-04-24T23:50:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f10ff7fe640 1 -- 172.21.15.16:0/1003654650 learned_addr learned my addr 172.21.15.16:0/1003654650 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f10fffff640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f110006e7c0 0x7f110013fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f11048bf640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1100145630 0x7f11000c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f10ff7fe640 1 -- 172.21.15.16:0/1003654650 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f110006e7c0 msgr2=0x7f110013fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f10ff7fe640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f110006e7c0 0x7f110013fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:50.651 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f10ff7fe640 1 -- 172.21.15.16:0/1003654650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1100145630 msgr2=0x7f11000c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.649+0000 7f10ff7fe640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1100145630 0x7f11000c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.650+0000 7f10ff7fe640 1 -- 172.21.15.16:0/1003654650 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11001e6f00 con 0x7f11000b54b0 2024-04-24T23:50:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.650+0000 7f11048bf640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1100145630 0x7f11000c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.650+0000 7f10ff7fe640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11000b54b0 0x7f11001e6540 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f10f000c430 tx=0x7f10f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.650+0000 7f10fcff9640 1 -- 172.21.15.16:0/1003654650 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0011790 con 0x7f11000b54b0 2024-04-24T23:50:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.650+0000 7f10fcff9640 1 -- 172.21.15.16:0/1003654650 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10f0017070 con 0x7f11000b54b0 2024-04-24T23:50:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.650+0000 7f1106322640 1 -- 172.21.15.16:0/1003654650 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11001e74c0 con 0x7f11000b54b0 2024-04-24T23:50:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.650+0000 7f10fcff9640 1 -- 172.21.15.16:0/1003654650 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10f0011bb0 con 0x7f11000b54b0 2024-04-24T23:50:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.650+0000 7f1106322640 1 -- 172.21.15.16:0/1003654650 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11001e7a80 con 0x7f11000b54b0 2024-04-24T23:50:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.651+0000 7f1106322640 1 -- 172.21.15.16:0/1003654650 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1100068f90 con 0x7f11000b54b0 2024-04-24T23:50:50.652 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.651+0000 7f10fcff9640 1 -- 172.21.15.16:0/1003654650 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f10f001c460 con 0x7f11000b54b0 2024-04-24T23:50:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.652+0000 7f10fcff9640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f10e006be30 0x7f10e006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:50.653 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.652+0000 7f10fcff9640 1 -- 172.21.15.16:0/1003654650 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f10f008abb0 con 0x7f11000b54b0 2024-04-24T23:50:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.652+0000 7f10fcff9640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.16:6824/2411432146,v1:172.21.15.16:6825/2411432146] conn(0x7f10e0072370 0x7f10e00747b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.652+0000 7f10fcff9640 1 -- 172.21.15.16:0/1003654650 --> [v2:172.21.15.16:6824/2411432146,v1:172.21.15.16:6825/2411432146] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10f0021e50 con 0x7f10e0072370 2024-04-24T23:50:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.652+0000 7f10fffff640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f10e006be30 0x7f10e006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.652+0000 7f10fcff9640 1 -- 172.21.15.16:0/1003654650 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f10f008d2b0 con 0x7f11000b54b0 2024-04-24T23:50:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.652+0000 7f11048bf640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.16:6824/2411432146,v1:172.21.15.16:6825/2411432146] conn(0x7f10e0072370 0x7f10e00747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.652+0000 7f10fffff640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f10e006be30 0x7f10e006e2f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f10e8034000 tx=0x7f10e802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.652+0000 7f11048bf640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.16:6824/2411432146,v1:172.21.15.16:6825/2411432146] conn(0x7f10e0072370 0x7f10e00747b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:50.654 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.653+0000 7f10fcff9640 1 -- 172.21.15.16:0/1003654650 <== osd.0 v2:172.21.15.16:6824/2411432146 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f10f0021e50 con 0x7f10e0072370 2024-04-24T23:50:50.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.666+0000 7f1106322640 1 -- 172.21.15.16:0/1003654650 --> [v2:172.21.15.16:6824/2411432146,v1:172.21.15.16:6825/2411432146] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1100070ac0 con 0x7f10e0072370 2024-04-24T23:50:50.666 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.666+0000 7f10fcff9640 1 -- 172.21.15.16:0/1003654650 <== osd.0 v2:172.21.15.16:6824/2411432146 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1100070ac0 con 0x7f10e0072370 2024-04-24T23:50:50.666 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:50:50.666 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:50:50.666 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:50:50.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.667+0000 7f1106322640 1 -- 172.21.15.16:0/1003654650 >> [v2:172.21.15.16:6824/2411432146,v1:172.21.15.16:6825/2411432146] conn(0x7f10e0072370 msgr2=0x7f10e00747b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:50.667 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.667+0000 7f1106322640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.16:6824/2411432146,v1:172.21.15.16:6825/2411432146] conn(0x7f10e0072370 0x7f10e00747b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.667+0000 7f1106322640 1 -- 172.21.15.16:0/1003654650 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f10e006be30 msgr2=0x7f10e006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.667+0000 7f1106322640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f10e006be30 0x7f10e006e2f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f10e8034000 tx=0x7f10e802e000 comp rx=0 tx=0).stop 2024-04-24T23:50:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.667+0000 7f1106322640 1 -- 172.21.15.16:0/1003654650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11000b54b0 msgr2=0x7f11001e6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.667+0000 7f1106322640 1 --2- 172.21.15.16:0/1003654650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11000b54b0 0x7f11001e6540 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f10f000c430 tx=0x7f10f000c8f0 comp rx=0 tx=0).stop 2024-04-24T23:50:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.667+0000 7f11048bf640 1 -- 172.21.15.16:0/1003654650 reap_dead start 2024-04-24T23:50:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.667+0000 7f1106322640 1 -- 172.21.15.16:0/1003654650 shutdown_connections 2024-04-24T23:50:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.667+0000 7f1106322640 1 -- 172.21.15.16:0/1003654650 >> 172.21.15.16:0/1003654650 conn(0x7f11000cfa20 msgr2=0x7f110015c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.667+0000 7f1106322640 1 -- 172.21.15.16:0/1003654650 shutdown_connections 2024-04-24T23:50:50.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.667+0000 7f1106322640 1 -- 172.21.15.16:0/1003654650 wait complete. 2024-04-24T23:50:50.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-04-24T23:50:50.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.779+0000 7f2232a4d640 1 Processor -- start 2024-04-24T23:50:50.779 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.779+0000 7f2232a4d640 1 -- start start 2024-04-24T23:50:50.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.780+0000 7f2232a4d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f222c0ce460 0x7f222c163a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:50.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.780+0000 7f2232a4d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f222c162c80 con 0x7f222c0ce080 2024-04-24T23:50:50.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.780+0000 7f2232a4d640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f222c162f60 con 0x7f222c0b3cc0 2024-04-24T23:50:50.780 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.780+0000 7f2232a4d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f222c163240 con 0x7f222c0ce460 2024-04-24T23:50:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.780+0000 7f222b7fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f222c0ce460 0x7f222c163a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.780+0000 7f222bfff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f222c0ce080 0x7f222c071c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57622/0 (socket says 172.21.15.16:57622) 2024-04-24T23:50:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.780+0000 7f222bfff640 1 -- 172.21.15.16:0/1480845953 learned_addr learned my addr 172.21.15.16:0/1480845953 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.780+0000 7f222b7fe640 1 -- 172.21.15.16:0/1480845953 >> v1:172.21.15.156:6789/0 conn(0x7f222c0b3cc0 legacy=0x7f222c0b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.780+0000 7f222b7fe640 1 -- 172.21.15.16:0/1480845953 >> v1:172.21.15.16:6789/0 conn(0x7f222c0ce080 legacy=0x7f222c071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.780+0000 7f222affd640 1 -- 172.21.15.16:0/1480845953 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4149616642 0 0) 0x7f222c162c80 con 0x7f222c0ce080 2024-04-24T23:50:50.781 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.781+0000 7f222b7fe640 1 -- 172.21.15.16:0/1480845953 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f222c163520 con 0x7f222c0ce460 2024-04-24T23:50:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.781+0000 7f222b7fe640 1 --2- 172.21.15.16:0/1480845953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f222c0ce460 0x7f222c163a60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f221c0097a0 tx=0x7f221c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eae4bd4ef3141e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.781+0000 7f222affd640 1 -- 172.21.15.16:0/1480845953 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221c037070 con 0x7f222c0ce460 2024-04-24T23:50:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.781+0000 7f222affd640 1 -- 172.21.15.16:0/1480845953 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f221c02ae50 con 0x7f222c0ce460 2024-04-24T23:50:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.781+0000 7f222affd640 1 -- 172.21.15.16:0/1480845953 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221c032660 con 0x7f222c0ce460 2024-04-24T23:50:50.782 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.782+0000 7f2232a4d640 1 -- 172.21.15.16:0/1480845953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f222c0ce460 msgr2=0x7f222c163a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.782+0000 7f2232a4d640 1 --2- 172.21.15.16:0/1480845953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f222c0ce460 0x7f222c163a60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f221c0097a0 tx=0x7f221c02a140 comp rx=0 tx=0).stop 2024-04-24T23:50:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.782+0000 7f2232a4d640 1 -- 172.21.15.16:0/1480845953 shutdown_connections 2024-04-24T23:50:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.782+0000 7f2232a4d640 1 --2- 172.21.15.16:0/1480845953 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f222c0ce460 0x7f222c163a60 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.782+0000 7f2232a4d640 1 -- 172.21.15.16:0/1480845953 >> 172.21.15.16:0/1480845953 conn(0x7f222c0b2000 msgr2=0x7f222c0cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.782+0000 7f2232a4d640 1 -- 172.21.15.16:0/1480845953 shutdown_connections 2024-04-24T23:50:50.783 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.782+0000 7f2232a4d640 1 -- 172.21.15.16:0/1480845953 wait complete. 2024-04-24T23:50:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.783+0000 7f2232a4d640 1 Processor -- start 2024-04-24T23:50:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.783+0000 7f2232a4d640 1 -- start start 2024-04-24T23:50:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f2232a4d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f222c0b3cc0 0x7f222c1f7640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f2232a4d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f222c0ce080 0x7f222c06da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f2232a4d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f222c0ce460 0x7f222c1573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f2232a4d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f222c162a20 con 0x7f222c0ce460 2024-04-24T23:50:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f2232a4d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f222c1628a0 con 0x7f222c0b3cc0 2024-04-24T23:50:50.785 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f2232a4d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f222c0b4170 con 0x7f222c0ce080 2024-04-24T23:50:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f2230fea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f222c0ce460 0x7f222c1573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f2230fea640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f222c0ce460 0x7f222c1573d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48954/0 (socket says 172.21.15.16:48954) 2024-04-24T23:50:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f2230fea640 1 -- 172.21.15.16:0/1989784022 learned_addr learned my addr 172.21.15.16:0/1989784022 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f222b7fe640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f222c0ce080 0x7f222c06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f222bfff640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f222c0b3cc0 0x7f222c1f7640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f2230fea640 1 -- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f222c0ce080 msgr2=0x7f222c06da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f2230fea640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f222c0ce080 0x7f222c06da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f2230fea640 1 -- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f222c0b3cc0 msgr2=0x7f222c1f7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f2230fea640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f222c0b3cc0 0x7f222c1f7640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f2230fea640 1 -- 172.21.15.16:0/1989784022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f222c155fc0 con 0x7f222c0ce460 2024-04-24T23:50:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f222b7fe640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f222c0ce080 0x7f222c06da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:50.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.784+0000 7f222bfff640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f222c0b3cc0 0x7f222c1f7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:50.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.785+0000 7f2230fea640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f222c0ce460 0x7f222c1573d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f222400c780 tx=0x7f222400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:50.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.785+0000 7f2228ff9640 1 -- 172.21.15.16:0/1989784022 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2224004b60 con 0x7f222c0ce460 2024-04-24T23:50:50.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.785+0000 7f2228ff9640 1 -- 172.21.15.16:0/1989784022 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2224004d00 con 0x7f222c0ce460 2024-04-24T23:50:50.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.785+0000 7f2232a4d640 1 -- 172.21.15.16:0/1989784022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f222c1562a0 con 0x7f222c0ce460 2024-04-24T23:50:50.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.785+0000 7f2228ff9640 1 -- 172.21.15.16:0/1989784022 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f222400aa90 con 0x7f222c0ce460 2024-04-24T23:50:50.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.785+0000 7f2232a4d640 1 -- 172.21.15.16:0/1989784022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f222c156860 con 0x7f222c0ce460 2024-04-24T23:50:50.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.786+0000 7f2232a4d640 1 -- 172.21.15.16:0/1989784022 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f222c0731f0 con 0x7f222c0ce460 2024-04-24T23:50:50.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.786+0000 7f2228ff9640 1 -- 172.21.15.16:0/1989784022 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f222400ac30 con 0x7f222c0ce460 2024-04-24T23:50:50.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.787+0000 7f2228ff9640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f221806be30 0x7f221806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:50.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.787+0000 7f2228ff9640 1 -- 172.21.15.16:0/1989784022 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f222408b050 con 0x7f222c0ce460 2024-04-24T23:50:50.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.787+0000 7f222bfff640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f221806be30 0x7f221806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:50.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.787+0000 7f222bfff640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f221806be30 0x7f221806e2f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2220000f60 tx=0x7f2220009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:50.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.791+0000 7f2228ff9640 1 -- 172.21.15.16:0/1989784022 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2224020070 con 0x7f222c0ce460 2024-04-24T23:50:50.979 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.979+0000 7f2232a4d640 1 -- 172.21.15.16:0/1989784022 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f222c04f5e0 con 0x7f221806be30 2024-04-24T23:50:50.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.980+0000 7f2228ff9640 1 -- 172.21.15.16:0/1989784022 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f222c04f5e0 con 0x7f221806be30 2024-04-24T23:50:50.980 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 0 to deep-scrub 2024-04-24T23:50:50.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.982+0000 7f2232a4d640 1 -- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f221806be30 msgr2=0x7f221806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:50.982 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.982+0000 7f2232a4d640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f221806be30 0x7f221806e2f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2220000f60 tx=0x7f2220009040 comp rx=0 tx=0).stop 2024-04-24T23:50:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.982+0000 7f2232a4d640 1 -- 172.21.15.16:0/1989784022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f222c0ce460 msgr2=0x7f222c1573d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.982+0000 7f2232a4d640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f222c0ce460 0x7f222c1573d0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f222400c780 tx=0x7f222400cc40 comp rx=0 tx=0).stop 2024-04-24T23:50:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.982+0000 7f2232a4d640 1 -- 172.21.15.16:0/1989784022 shutdown_connections 2024-04-24T23:50:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.982+0000 7f2232a4d640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f221806be30 0x7f221806e2f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.982+0000 7f2232a4d640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f222c0ce460 0x7f222c1573d0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.982+0000 7f2232a4d640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f222c0ce080 0x7f222c06da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.982+0000 7f2232a4d640 1 --2- 172.21.15.16:0/1989784022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f222c0b3cc0 0x7f222c1f7640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.983+0000 7f2232a4d640 1 -- 172.21.15.16:0/1989784022 >> 172.21.15.16:0/1989784022 conn(0x7f222c0b2000 msgr2=0x7f222c165e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.983+0000 7f2232a4d640 1 -- 172.21.15.16:0/1989784022 shutdown_connections 2024-04-24T23:50:50.983 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:50.983+0000 7f2232a4d640 1 -- 172.21.15.16:0/1989784022 wait complete. 2024-04-24T23:50:50.993 INFO:tasks.ceph:Scrubbing osd.1 2024-04-24T23:50:50.993 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:50:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.098+0000 7ff6da24d640 1 Processor -- start 2024-04-24T23:50:51.098 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.098+0000 7ff6da24d640 1 -- start start 2024-04-24T23:50:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6da24d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff6d415df90 0x7ff6d4072b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6da24d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6d406f0d0 0x7ff6d4073050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6da24d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6d415e370 con 0x7ff6d40cfc10 2024-04-24T23:50:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6da24d640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7ff6d41627c0 con 0x7ff6d415df90 2024-04-24T23:50:51.099 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6da24d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff6d406ec10 con 0x7ff6d406f0d0 2024-04-24T23:50:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6d37fe640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff6d415df90 0x7ff6d4072b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6d2ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff6d40cfc10 0x7ff6d40ad8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57626/0 (socket says 172.21.15.16:57626) 2024-04-24T23:50:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6d2ffd640 1 -- 172.21.15.16:0/1851046895 learned_addr learned my addr 172.21.15.16:0/1851046895 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6d3fff640 1 --2- 172.21.15.16:0/1851046895 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6d406f0d0 0x7ff6d4073050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6d27fc640 1 -- 172.21.15.16:0/1851046895 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1625301718 0 0) 0x7ff6d415e370 con 0x7ff6d40cfc10 2024-04-24T23:50:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6d27fc640 1 -- 172.21.15.16:0/1851046895 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff6bc003520 con 0x7ff6d40cfc10 2024-04-24T23:50:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6d37fe640 1 -- 172.21.15.16:0/1851046895 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6d406f0d0 msgr2=0x7ff6d4073050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6d37fe640 1 --2- 172.21.15.16:0/1851046895 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6d406f0d0 0x7ff6d4073050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6d37fe640 1 -- 172.21.15.16:0/1851046895 >> v1:172.21.15.16:6789/0 conn(0x7ff6d40cfc10 legacy=0x7ff6d40ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.100 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.099+0000 7ff6d37fe640 1 -- 172.21.15.16:0/1851046895 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d40cf780 con 0x7ff6d415df90 2024-04-24T23:50:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.100+0000 7ff6d37fe640 1 --2- 172.21.15.16:0/1851046895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d415df90 0x7ff6d4072b10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff6c40097d0 tx=0x7ff6c402a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c2c8c2770991878 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.100+0000 7ff6d27fc640 1 -- 172.21.15.16:0/1851046895 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4037070 con 0x7ff6d415df90 2024-04-24T23:50:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.100+0000 7ff6d27fc640 1 -- 172.21.15.16:0/1851046895 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6c4032440 con 0x7ff6d415df90 2024-04-24T23:50:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.100+0000 7ff6d27fc640 1 -- 172.21.15.16:0/1851046895 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4032810 con 0x7ff6d415df90 2024-04-24T23:50:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.101+0000 7ff6da24d640 1 -- 172.21.15.16:0/1851046895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d415df90 msgr2=0x7ff6d4072b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.101+0000 7ff6da24d640 1 --2- 172.21.15.16:0/1851046895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d415df90 0x7ff6d4072b10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7ff6c40097d0 tx=0x7ff6c402a1c0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.101+0000 7ff6da24d640 1 -- 172.21.15.16:0/1851046895 shutdown_connections 2024-04-24T23:50:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.101+0000 7ff6da24d640 1 --2- 172.21.15.16:0/1851046895 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6d406f0d0 0x7ff6d4073050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.101+0000 7ff6da24d640 1 --2- 172.21.15.16:0/1851046895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d415df90 0x7ff6d4072b10 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.101+0000 7ff6da24d640 1 -- 172.21.15.16:0/1851046895 >> 172.21.15.16:0/1851046895 conn(0x7ff6d40d0aa0 msgr2=0x7ff6d40d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.101+0000 7ff6da24d640 1 -- 172.21.15.16:0/1851046895 shutdown_connections 2024-04-24T23:50:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.101+0000 7ff6da24d640 1 -- 172.21.15.16:0/1851046895 wait complete. 2024-04-24T23:50:51.102 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.102+0000 7ff6da24d640 1 Processor -- start 2024-04-24T23:50:51.103 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.102+0000 7ff6da24d640 1 -- start start 2024-04-24T23:50:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6da24d640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6d406f0d0 0x7ff6d4143c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6da24d640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6d40cfc10 0x7ff6d41422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6da24d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d415df90 0x7ff6d4151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6da24d640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d4162170 con 0x7ff6d415df90 2024-04-24T23:50:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6da24d640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d4161e90 con 0x7ff6d406f0d0 2024-04-24T23:50:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6da24d640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7ff6d4162450 con 0x7ff6d40cfc10 2024-04-24T23:50:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d415df90 0x7ff6d4151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6d3fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d415df90 0x7ff6d4151fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48962/0 (socket says 172.21.15.16:48962) 2024-04-24T23:50:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6d3fff640 1 -- 172.21.15.16:0/1967344838 learned_addr learned my addr 172.21.15.16:0/1967344838 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6d2ffd640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6d40cfc10 0x7ff6d41422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6d37fe640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6d406f0d0 0x7ff6d4143c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6d3fff640 1 -- 172.21.15.16:0/1967344838 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6d40cfc10 msgr2=0x7ff6d41422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6d3fff640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6d40cfc10 0x7ff6d41422d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6d3fff640 1 -- 172.21.15.16:0/1967344838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6d406f0d0 msgr2=0x7ff6d4143c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6d3fff640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff6d406f0d0 0x7ff6d4143c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6d3fff640 1 -- 172.21.15.16:0/1967344838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d4142be0 con 0x7ff6d415df90 2024-04-24T23:50:51.106 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6d2ffd640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7ff6d40cfc10 0x7ff6d41422d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.103+0000 7ff6d3fff640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d415df90 0x7ff6d4151fe0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc007e40 tx=0x7ff6cc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.104+0000 7ff6bbfff640 1 -- 172.21.15.16:0/1967344838 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc014620 con 0x7ff6d415df90 2024-04-24T23:50:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.104+0000 7ff6bbfff640 1 -- 172.21.15.16:0/1967344838 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6cc019070 con 0x7ff6d415df90 2024-04-24T23:50:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.104+0000 7ff6da24d640 1 -- 172.21.15.16:0/1967344838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d4142ec0 con 0x7ff6d415df90 2024-04-24T23:50:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.104+0000 7ff6bbfff640 1 -- 172.21.15.16:0/1967344838 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc014a20 con 0x7ff6d415df90 2024-04-24T23:50:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.104+0000 7ff6da24d640 1 -- 172.21.15.16:0/1967344838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6d4143480 con 0x7ff6d415df90 2024-04-24T23:50:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.104+0000 7ff6da24d640 1 -- 172.21.15.16:0/1967344838 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff6d4068f90 con 0x7ff6d415df90 2024-04-24T23:50:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.105+0000 7ff6bbfff640 1 -- 172.21.15.16:0/1967344838 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff6cc014bc0 con 0x7ff6d415df90 2024-04-24T23:50:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.105+0000 7ff6bbfff640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff6c006be60 0x7ff6c006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.106+0000 7ff6bbfff640 1 -- 172.21.15.16:0/1967344838 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7ff6cc08b600 con 0x7ff6d415df90 2024-04-24T23:50:51.107 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.106+0000 7ff6bbfff640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.16:6800/2543873173,v1:172.21.15.16:6801/2543873173] conn(0x7ff6c00723a0 0x7ff6c00747e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.106+0000 7ff6d37fe640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff6c006be60 0x7ff6c006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.106+0000 7ff6bbfff640 1 -- 172.21.15.16:0/1967344838 --> [v2:172.21.15.16:6800/2543873173,v1:172.21.15.16:6801/2543873173] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6c0074eb0 con 0x7ff6c00723a0 2024-04-24T23:50:51.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.106+0000 7ff6bbfff640 1 -- 172.21.15.16:0/1967344838 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7ff6cc092050 con 0x7ff6d415df90 2024-04-24T23:50:51.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.106+0000 7ff6d2ffd640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.16:6800/2543873173,v1:172.21.15.16:6801/2543873173] conn(0x7ff6c00723a0 0x7ff6c00747e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.106+0000 7ff6d2ffd640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.16:6800/2543873173,v1:172.21.15.16:6801/2543873173] conn(0x7ff6c00723a0 0x7ff6c00747e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:51.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.106+0000 7ff6d37fe640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff6c006be60 0x7ff6c006e320 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4034000 tx=0x7ff6c402e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:51.108 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.107+0000 7ff6bbfff640 1 -- 172.21.15.16:0/1967344838 <== osd.1 v2:172.21.15.16:6800/2543873173 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff6c0074eb0 con 0x7ff6c00723a0 2024-04-24T23:50:51.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.120+0000 7ff6da24d640 1 -- 172.21.15.16:0/1967344838 --> [v2:172.21.15.16:6800/2543873173,v1:172.21.15.16:6801/2543873173] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff6d40ccaf0 con 0x7ff6c00723a0 2024-04-24T23:50:51.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.120+0000 7ff6bbfff640 1 -- 172.21.15.16:0/1967344838 <== osd.1 v2:172.21.15.16:6800/2543873173 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff6d40ccaf0 con 0x7ff6c00723a0 2024-04-24T23:50:51.120 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:50:51.121 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:50:51.121 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:50:51.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.120+0000 7ff6da24d640 1 -- 172.21.15.16:0/1967344838 >> [v2:172.21.15.16:6800/2543873173,v1:172.21.15.16:6801/2543873173] conn(0x7ff6c00723a0 msgr2=0x7ff6c00747e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.120+0000 7ff6da24d640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.16:6800/2543873173,v1:172.21.15.16:6801/2543873173] conn(0x7ff6c00723a0 0x7ff6c00747e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.120+0000 7ff6da24d640 1 -- 172.21.15.16:0/1967344838 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff6c006be60 msgr2=0x7ff6c006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.120+0000 7ff6da24d640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7ff6c006be60 0x7ff6c006e320 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4034000 tx=0x7ff6c402e000 comp rx=0 tx=0).stop 2024-04-24T23:50:51.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.120+0000 7ff6da24d640 1 -- 172.21.15.16:0/1967344838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d415df90 msgr2=0x7ff6d4151fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.120+0000 7ff6da24d640 1 --2- 172.21.15.16:0/1967344838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff6d415df90 0x7ff6d4151fe0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc007e40 tx=0x7ff6cc00e4b0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.120+0000 7ff6d3fff640 1 -- 172.21.15.16:0/1967344838 reap_dead start 2024-04-24T23:50:51.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.121+0000 7ff6da24d640 1 -- 172.21.15.16:0/1967344838 shutdown_connections 2024-04-24T23:50:51.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.121+0000 7ff6da24d640 1 -- 172.21.15.16:0/1967344838 >> 172.21.15.16:0/1967344838 conn(0x7ff6d40d0aa0 msgr2=0x7ff6d4071a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:51.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.121+0000 7ff6da24d640 1 -- 172.21.15.16:0/1967344838 shutdown_connections 2024-04-24T23:50:51.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.121+0000 7ff6da24d640 1 -- 172.21.15.16:0/1967344838 wait complete. 2024-04-24T23:50:51.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-04-24T23:50:51.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.233+0000 7fbab56b3640 1 Processor -- start 2024-04-24T23:50:51.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.233+0000 7fbab56b3640 1 -- start start 2024-04-24T23:50:51.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.234+0000 7fbab56b3640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbab015df40 0x7fbab0072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.234+0000 7fbab56b3640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbab00727d0 0x7fbab0161c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.234+0000 7fbab56b3640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbab00b4c00 con 0x7fbab01621b0 2024-04-24T23:50:51.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.234+0000 7fbab56b3640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fbab0072c30 con 0x7fbab00727d0 2024-04-24T23:50:51.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.234+0000 7fbab56b3640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fbab00ae2b0 con 0x7fbab015df40 2024-04-24T23:50:51.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.234+0000 7fbaaeffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbab015df40 0x7fbab0072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.234+0000 7fbaae7fc640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fbab00727d0 0x7fbab0161c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.234+0000 7fbaaeffd640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbab015df40 0x7fbab0072290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3301/0 says I am v2:172.21.15.16:52924/0 (socket says 172.21.15.16:52924) 2024-04-24T23:50:51.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.234+0000 7fbaaeffd640 1 -- 172.21.15.16:0/4023612592 learned_addr learned my addr 172.21.15.16:0/4023612592 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:51.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.235+0000 7fbaaeffd640 1 -- 172.21.15.16:0/4023612592 >> v2:172.21.15.156:3300/0 conn(0x7fbab00727d0 msgr2=0x7fbab0161c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:51.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.235+0000 7fbaaeffd640 1 --2- 172.21.15.16:0/4023612592 >> v2:172.21.15.156:3300/0 conn(0x7fbab00727d0 0x7fbab0161c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.235+0000 7fbaaeffd640 1 -- 172.21.15.16:0/4023612592 >> v1:172.21.15.156:6789/0 conn(0x7fbab01621b0 legacy=0x7fbab00aec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.235+0000 7fbaaeffd640 1 -- 172.21.15.16:0/4023612592 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab0162a10 con 0x7fbab015df40 2024-04-24T23:50:51.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.235+0000 7fbaadffb640 1 -- 172.21.15.16:0/4023612592 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 522507290 0 0) 0x7fbab00b4c00 con 0x7fbab01621b0 2024-04-24T23:50:51.235 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.235+0000 7fbaae7fc640 1 --2- 172.21.15.16:0/4023612592 >> v2:172.21.15.156:3300/0 conn(0x7fbab00727d0 0x7fbab0161c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:50:51.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.235+0000 7fbaaeffd640 1 --2- 172.21.15.16:0/4023612592 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbab015df40 0x7fbab0072290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8009870 tx=0x7fbaa802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94ec1a165d9b724e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:51.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.236+0000 7fbaadffb640 1 -- 172.21.15.16:0/4023612592 <== mon.2 v2:172.21.15.156:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8037070 con 0x7fbab015df40 2024-04-24T23:50:51.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.236+0000 7fbaadffb640 1 -- 172.21.15.16:0/4023612592 <== mon.2 v2:172.21.15.156:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaa8032440 con 0x7fbab015df40 2024-04-24T23:50:51.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.236+0000 7fbaadffb640 1 -- 172.21.15.16:0/4023612592 <== mon.2 v2:172.21.15.156:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8032810 con 0x7fbab015df40 2024-04-24T23:50:51.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.236+0000 7fbab56b3640 1 -- 172.21.15.16:0/4023612592 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbab015df40 msgr2=0x7fbab0072290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.236+0000 7fbab56b3640 1 --2- 172.21.15.16:0/4023612592 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbab015df40 0x7fbab0072290 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8009870 tx=0x7fbaa802a270 comp rx=0 tx=0).stop 2024-04-24T23:50:51.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.236+0000 7fbab56b3640 1 -- 172.21.15.16:0/4023612592 shutdown_connections 2024-04-24T23:50:51.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.236+0000 7fbab56b3640 1 --2- 172.21.15.16:0/4023612592 >> v2:172.21.15.156:3300/0 conn(0x7fbab00727d0 0x7fbab0161c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.236+0000 7fbab56b3640 1 --2- 172.21.15.16:0/4023612592 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbab015df40 0x7fbab0072290 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.236+0000 7fbab56b3640 1 -- 172.21.15.16:0/4023612592 >> 172.21.15.16:0/4023612592 conn(0x7fbab00d0260 msgr2=0x7fbab00d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:51.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.236+0000 7fbab56b3640 1 -- 172.21.15.16:0/4023612592 shutdown_connections 2024-04-24T23:50:51.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.236+0000 7fbab56b3640 1 -- 172.21.15.16:0/4023612592 wait complete. 2024-04-24T23:50:51.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.237+0000 7fbab56b3640 1 Processor -- start 2024-04-24T23:50:51.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.238+0000 7fbab56b3640 1 -- start start 2024-04-24T23:50:51.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.238+0000 7fbab56b3640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab00727d0 0x7fbab015b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.238+0000 7fbab56b3640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbab015df40 0x7fbab014e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.238+0000 7fbab56b3640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbab01621b0 0x7fbab0158930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.238+0000 7fbab56b3640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fbab00adf40 con 0x7fbab00727d0 2024-04-24T23:50:51.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.238+0000 7fbab56b3640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbab00adc60 con 0x7fbab015df40 2024-04-24T23:50:51.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.238+0000 7fbab56b3640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fbab00ae740 con 0x7fbab01621b0 2024-04-24T23:50:51.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.238+0000 7fbaaeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab00727d0 0x7fbab015b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.238+0000 7fbaaeffd640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab00727d0 0x7fbab015b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48970/0 (socket says 172.21.15.16:48970) 2024-04-24T23:50:51.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.238+0000 7fbaaeffd640 1 -- 172.21.15.16:0/1405515662 learned_addr learned my addr 172.21.15.16:0/1405515662 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:51.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.239+0000 7fbaae7fc640 1 --2- 172.21.15.16:0/1405515662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbab015df40 0x7fbab014e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.239+0000 7fbaaf7fe640 1 --2- 172.21.15.16:0/1405515662 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbab01621b0 0x7fbab0158930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.239+0000 7fbaaeffd640 1 -- 172.21.15.16:0/1405515662 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbab01621b0 msgr2=0x7fbab0158930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.239+0000 7fbaaeffd640 1 --2- 172.21.15.16:0/1405515662 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbab01621b0 0x7fbab0158930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.239+0000 7fbaaeffd640 1 -- 172.21.15.16:0/1405515662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbab015df40 msgr2=0x7fbab014e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.239+0000 7fbaaeffd640 1 --2- 172.21.15.16:0/1405515662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbab015df40 0x7fbab014e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.239+0000 7fbaaeffd640 1 -- 172.21.15.16:0/1405515662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbab0072580 con 0x7fbab00727d0 2024-04-24T23:50:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.239+0000 7fbaaeffd640 1 --2- 172.21.15.16:0/1405515662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab00727d0 0x7fbab015b860 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8035040 tx=0x7fbaa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.239+0000 7fbaa77fe640 1 -- 172.21.15.16:0/1405515662 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8037040 con 0x7fbab00727d0 2024-04-24T23:50:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.239+0000 7fbaa77fe640 1 -- 172.21.15.16:0/1405515662 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbaa8004030 con 0x7fbab00727d0 2024-04-24T23:50:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.239+0000 7fbaa77fe640 1 -- 172.21.15.16:0/1405515662 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaa8004420 con 0x7fbab00727d0 2024-04-24T23:50:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.239+0000 7fbab56b3640 1 -- 172.21.15.16:0/1405515662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbab015bef0 con 0x7fbab00727d0 2024-04-24T23:50:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.239+0000 7fbab56b3640 1 -- 172.21.15.16:0/1405515662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbab015c4b0 con 0x7fbab00727d0 2024-04-24T23:50:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.240+0000 7fbab56b3640 1 -- 172.21.15.16:0/1405515662 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbab006e5c0 con 0x7fbab00727d0 2024-04-24T23:50:51.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.241+0000 7fbaa77fe640 1 -- 172.21.15.16:0/1405515662 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbaa803a070 con 0x7fbab00727d0 2024-04-24T23:50:51.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.241+0000 7fbaa77fe640 1 --2- 172.21.15.16:0/1405515662 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fba9006be30 0x7fba9006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.241+0000 7fbaa77fe640 1 -- 172.21.15.16:0/1405515662 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7fbaa80ab260 con 0x7fbab00727d0 2024-04-24T23:50:51.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.241+0000 7fbaae7fc640 1 --2- 172.21.15.16:0/1405515662 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fba9006be30 0x7fba9006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.242+0000 7fbaae7fc640 1 --2- 172.21.15.16:0/1405515662 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fba9006be30 0x7fba9006e2f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fba98009560 tx=0x7fba98009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:51.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.245+0000 7fbaa77fe640 1 -- 172.21.15.16:0/1405515662 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fbaa8041020 con 0x7fbab00727d0 2024-04-24T23:50:51.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.426+0000 7fbab56b3640 1 -- 172.21.15.16:0/1405515662 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7fbab00d0040 con 0x7fba9006be30 2024-04-24T23:50:51.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.426+0000 7fbaa77fe640 1 -- 172.21.15.16:0/1405515662 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbab00d0040 con 0x7fba9006be30 2024-04-24T23:50:51.426 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 1 to deep-scrub 2024-04-24T23:50:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.428+0000 7fbab56b3640 1 -- 172.21.15.16:0/1405515662 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fba9006be30 msgr2=0x7fba9006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.428+0000 7fbab56b3640 1 --2- 172.21.15.16:0/1405515662 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fba9006be30 0x7fba9006e2f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fba98009560 tx=0x7fba98009190 comp rx=0 tx=0).stop 2024-04-24T23:50:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.428+0000 7fbab56b3640 1 -- 172.21.15.16:0/1405515662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab00727d0 msgr2=0x7fbab015b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.428+0000 7fbab56b3640 1 --2- 172.21.15.16:0/1405515662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab00727d0 0x7fbab015b860 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8035040 tx=0x7fbaa80046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.428+0000 7fbab56b3640 1 -- 172.21.15.16:0/1405515662 shutdown_connections 2024-04-24T23:50:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.428+0000 7fbab56b3640 1 --2- 172.21.15.16:0/1405515662 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fba9006be30 0x7fba9006e2f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.428 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.428+0000 7fbab56b3640 1 --2- 172.21.15.16:0/1405515662 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fbab01621b0 0x7fbab0158930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.428+0000 7fbab56b3640 1 --2- 172.21.15.16:0/1405515662 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbab015df40 0x7fbab014e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.428+0000 7fbab56b3640 1 --2- 172.21.15.16:0/1405515662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbab00727d0 0x7fbab015b860 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.428+0000 7fbab56b3640 1 -- 172.21.15.16:0/1405515662 >> 172.21.15.16:0/1405515662 conn(0x7fbab00d0260 msgr2=0x7fbab015ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.428+0000 7fbab56b3640 1 -- 172.21.15.16:0/1405515662 shutdown_connections 2024-04-24T23:50:51.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.428+0000 7fbab56b3640 1 -- 172.21.15.16:0/1405515662 wait complete. 2024-04-24T23:50:51.439 INFO:tasks.ceph:Scrubbing osd.2 2024-04-24T23:50:51.439 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:50:51.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.543+0000 7fd181561640 1 Processor -- start 2024-04-24T23:50:51.543 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.544+0000 7fd181561640 1 -- start start 2024-04-24T23:50:51.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.544+0000 7fd181561640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fd17c15e2b0 0x7fd17c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.544+0000 7fd181561640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd17c1627a0 0x7fd17c0cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.544+0000 7fd181561640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd17c0cd900 0x7fd17c0b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.545 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.544+0000 7fd181561640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fd17c15e690 con 0x7fd17c1627a0 2024-04-24T23:50:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.544+0000 7fd181561640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fd17c0ccf30 con 0x7fd17c15e2b0 2024-04-24T23:50:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.544+0000 7fd181561640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fd17c0cc670 con 0x7fd17c0cd900 2024-04-24T23:50:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.544+0000 7fd17a7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd17c1627a0 0x7fd17c0cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.545+0000 7fd17a7fc640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd17c1627a0 0x7fd17c0cd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48978/0 (socket says 172.21.15.16:48978) 2024-04-24T23:50:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.545+0000 7fd17a7fc640 1 -- 172.21.15.16:0/1762073160 learned_addr learned my addr 172.21.15.16:0/1762073160 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.545+0000 7fd17b7fe640 1 --2- 172.21.15.16:0/1762073160 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd17c0cd900 0x7fd17c0b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.545+0000 7fd17affd640 1 --2- 172.21.15.16:0/1762073160 >> v2:172.21.15.156:3300/0 conn(0x7fd17c15e2b0 0x7fd17c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.545+0000 7fd17a7fc640 1 -- 172.21.15.16:0/1762073160 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd17c0cd900 msgr2=0x7fd17c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.545+0000 7fd17a7fc640 1 --2- 172.21.15.16:0/1762073160 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd17c0cd900 0x7fd17c0b4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.545+0000 7fd17a7fc640 1 -- 172.21.15.16:0/1762073160 >> v2:172.21.15.156:3300/0 conn(0x7fd17c15e2b0 msgr2=0x7fd17c0711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:51.546 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.545+0000 7fd17a7fc640 1 --2- 172.21.15.16:0/1762073160 >> v2:172.21.15.156:3300/0 conn(0x7fd17c15e2b0 0x7fd17c0711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.545+0000 7fd17a7fc640 1 -- 172.21.15.16:0/1762073160 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17c0cc8e0 con 0x7fd17c1627a0 2024-04-24T23:50:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.546+0000 7fd17a7fc640 1 --2- 172.21.15.16:0/1762073160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c1627a0 0x7fd17c0cd3c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd17c0725a0 tx=0x7fd16402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ec5599c41e8127d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.546+0000 7fd179ffb640 1 -- 172.21.15.16:0/1762073160 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164037070 con 0x7fd17c1627a0 2024-04-24T23:50:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.546+0000 7fd179ffb640 1 -- 172.21.15.16:0/1762073160 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd164032440 con 0x7fd17c1627a0 2024-04-24T23:50:51.547 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.546+0000 7fd179ffb640 1 -- 172.21.15.16:0/1762073160 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd164032810 con 0x7fd17c1627a0 2024-04-24T23:50:51.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.546+0000 7fd181561640 1 -- 172.21.15.16:0/1762073160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c1627a0 msgr2=0x7fd17c0cd3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.546+0000 7fd181561640 1 --2- 172.21.15.16:0/1762073160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c1627a0 0x7fd17c0cd3c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fd17c0725a0 tx=0x7fd16402a210 comp rx=0 tx=0).stop 2024-04-24T23:50:51.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.546+0000 7fd181561640 1 -- 172.21.15.16:0/1762073160 shutdown_connections 2024-04-24T23:50:51.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.546+0000 7fd181561640 1 --2- 172.21.15.16:0/1762073160 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd17c0cd900 0x7fd17c0b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.546+0000 7fd181561640 1 --2- 172.21.15.16:0/1762073160 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c1627a0 0x7fd17c0cd3c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.546+0000 7fd181561640 1 --2- 172.21.15.16:0/1762073160 >> v2:172.21.15.156:3300/0 conn(0x7fd17c15e2b0 0x7fd17c0711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.546+0000 7fd181561640 1 -- 172.21.15.16:0/1762073160 >> 172.21.15.16:0/1762073160 conn(0x7fd17c0d1bc0 msgr2=0x7fd17c0d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.546+0000 7fd181561640 1 -- 172.21.15.16:0/1762073160 shutdown_connections 2024-04-24T23:50:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.546+0000 7fd181561640 1 -- 172.21.15.16:0/1762073160 wait complete. 2024-04-24T23:50:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.548+0000 7fd181561640 1 Processor -- start 2024-04-24T23:50:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.548+0000 7fd181561640 1 -- start start 2024-04-24T23:50:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.548+0000 7fd181561640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd17c15e2b0 0x7fd17c0c9670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.548+0000 7fd181561640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd17c0cb850 0x7fd17c0b0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.548+0000 7fd181561640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0d0510 0x7fd17c0d09a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.548+0000 7fd181561640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fd17c0cdf30 con 0x7fd17c0d0510 2024-04-24T23:50:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.548+0000 7fd181561640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd17c162c30 con 0x7fd17c15e2b0 2024-04-24T23:50:51.553 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.548+0000 7fd181561640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fd17c0b4f90 con 0x7fd17c0cb850 2024-04-24T23:50:51.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.548+0000 7fd17b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0d0510 0x7fd17c0d09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd17b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0d0510 0x7fd17c0d09a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48984/0 (socket says 172.21.15.16:48984) 2024-04-24T23:50:51.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd17b7fe640 1 -- 172.21.15.16:0/2389268664 learned_addr learned my addr 172.21.15.16:0/2389268664 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:51.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd17affd640 1 --2- 172.21.15.16:0/2389268664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd17c15e2b0 0x7fd17c0c9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd17a7fc640 1 --2- 172.21.15.16:0/2389268664 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd17c0cb850 0x7fd17c0b0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd17b7fe640 1 -- 172.21.15.16:0/2389268664 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd17c0cb850 msgr2=0x7fd17c0b0c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd17b7fe640 1 --2- 172.21.15.16:0/2389268664 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fd17c0cb850 0x7fd17c0b0c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd17b7fe640 1 -- 172.21.15.16:0/2389268664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd17c15e2b0 msgr2=0x7fd17c0c9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.554 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd17b7fe640 1 --2- 172.21.15.16:0/2389268664 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd17c15e2b0 0x7fd17c0c9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd17b7fe640 1 -- 172.21.15.16:0/2389268664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd17c0c9ca0 con 0x7fd17c0d0510 2024-04-24T23:50:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd17b7fe640 1 --2- 172.21.15.16:0/2389268664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0d0510 0x7fd17c0d09a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00eed0 tx=0x7fd16c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd1737fe640 1 -- 172.21.15.16:0/2389268664 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c018020 con 0x7fd17c0d0510 2024-04-24T23:50:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd1737fe640 1 -- 172.21.15.16:0/2389268664 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd16c014400 con 0x7fd17c0d0510 2024-04-24T23:50:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd181561640 1 -- 172.21.15.16:0/2389268664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd17c0c9f80 con 0x7fd17c0d0510 2024-04-24T23:50:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd1737fe640 1 -- 172.21.15.16:0/2389268664 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd16c0147f0 con 0x7fd17c0d0510 2024-04-24T23:50:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.549+0000 7fd181561640 1 -- 172.21.15.16:0/2389268664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd17c0ca540 con 0x7fd17c0d0510 2024-04-24T23:50:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.550+0000 7fd181561640 1 -- 172.21.15.16:0/2389268664 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd17c0b4660 con 0x7fd17c0d0510 2024-04-24T23:50:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.551+0000 7fd1737fe640 1 -- 172.21.15.16:0/2389268664 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd16c005d10 con 0x7fd17c0d0510 2024-04-24T23:50:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.551+0000 7fd1737fe640 1 --2- 172.21.15.16:0/2389268664 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd15c06be60 0x7fd15c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.551+0000 7fd1737fe640 1 -- 172.21.15.16:0/2389268664 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7fd16c08b5f0 con 0x7fd17c0d0510 2024-04-24T23:50:51.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.551+0000 7fd1737fe640 1 --2- 172.21.15.16:0/2389268664 >> [v2:172.21.15.16:6802/4175732037,v1:172.21.15.16:6804/4175732037] conn(0x7fd15c0723a0 0x7fd15c0747e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.551+0000 7fd1737fe640 1 -- 172.21.15.16:0/2389268664 --> [v2:172.21.15.16:6802/4175732037,v1:172.21.15.16:6804/4175732037] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd15c074eb0 con 0x7fd15c0723a0 2024-04-24T23:50:51.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.551+0000 7fd1737fe640 1 -- 172.21.15.16:0/2389268664 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fd16c092050 con 0x7fd17c0d0510 2024-04-24T23:50:51.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.551+0000 7fd17affd640 1 --2- 172.21.15.16:0/2389268664 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd15c06be60 0x7fd15c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.551+0000 7fd17a7fc640 1 --2- 172.21.15.16:0/2389268664 >> [v2:172.21.15.16:6802/4175732037,v1:172.21.15.16:6804/4175732037] conn(0x7fd15c0723a0 0x7fd15c0747e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.552+0000 7fd17a7fc640 1 --2- 172.21.15.16:0/2389268664 >> [v2:172.21.15.16:6802/4175732037,v1:172.21.15.16:6804/4175732037] conn(0x7fd15c0723a0 0x7fd15c0747e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:51.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.552+0000 7fd17affd640 1 --2- 172.21.15.16:0/2389268664 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd15c06be60 0x7fd15c06e320 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd174009b40 tx=0x7fd174009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:51.556 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.552+0000 7fd1737fe640 1 -- 172.21.15.16:0/2389268664 <== osd.2 v2:172.21.15.16:6802/4175732037 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd15c074eb0 con 0x7fd15c0723a0 2024-04-24T23:50:51.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.565+0000 7fd181561640 1 -- 172.21.15.16:0/2389268664 --> [v2:172.21.15.16:6802/4175732037,v1:172.21.15.16:6804/4175732037] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fd17c068f90 con 0x7fd15c0723a0 2024-04-24T23:50:51.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.566+0000 7fd1737fe640 1 -- 172.21.15.16:0/2389268664 <== osd.2 v2:172.21.15.16:6802/4175732037 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fd17c068f90 con 0x7fd15c0723a0 2024-04-24T23:50:51.566 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:50:51.566 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:50:51.566 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:50:51.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.566+0000 7fd181561640 1 -- 172.21.15.16:0/2389268664 >> [v2:172.21.15.16:6802/4175732037,v1:172.21.15.16:6804/4175732037] conn(0x7fd15c0723a0 msgr2=0x7fd15c0747e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.566+0000 7fd181561640 1 --2- 172.21.15.16:0/2389268664 >> [v2:172.21.15.16:6802/4175732037,v1:172.21.15.16:6804/4175732037] conn(0x7fd15c0723a0 0x7fd15c0747e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.566+0000 7fd181561640 1 -- 172.21.15.16:0/2389268664 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd15c06be60 msgr2=0x7fd15c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.566+0000 7fd181561640 1 --2- 172.21.15.16:0/2389268664 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fd15c06be60 0x7fd15c06e320 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fd174009b40 tx=0x7fd174009190 comp rx=0 tx=0).stop 2024-04-24T23:50:51.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.566+0000 7fd181561640 1 -- 172.21.15.16:0/2389268664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0d0510 msgr2=0x7fd17c0d09a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.566+0000 7fd181561640 1 --2- 172.21.15.16:0/2389268664 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd17c0d0510 0x7fd17c0d09a0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fd16c00eed0 tx=0x7fd16c00b480 comp rx=0 tx=0).stop 2024-04-24T23:50:51.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.566+0000 7fd17b7fe640 1 -- 172.21.15.16:0/2389268664 reap_dead start 2024-04-24T23:50:51.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.566+0000 7fd181561640 1 -- 172.21.15.16:0/2389268664 shutdown_connections 2024-04-24T23:50:51.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.566+0000 7fd181561640 1 -- 172.21.15.16:0/2389268664 >> 172.21.15.16:0/2389268664 conn(0x7fd17c0d1bc0 msgr2=0x7fd17c0b1f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:51.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.566+0000 7fd181561640 1 -- 172.21.15.16:0/2389268664 shutdown_connections 2024-04-24T23:50:51.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.566+0000 7fd181561640 1 -- 172.21.15.16:0/2389268664 wait complete. 2024-04-24T23:50:51.576 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-04-24T23:50:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.678+0000 7f2429eae640 1 Processor -- start 2024-04-24T23:50:51.678 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.678+0000 7f2429eae640 1 -- start start 2024-04-24T23:50:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.679+0000 7f2429eae640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f242415e020 0x7f24240cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.679+0000 7f2429eae640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24240b2bb0 0x7f24240cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.679+0000 7f2429eae640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24240b3390 con 0x7f24240cfcf0 2024-04-24T23:50:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.679+0000 7f2429eae640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f242406bfc0 con 0x7f242415e020 2024-04-24T23:50:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.679+0000 7f2429eae640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f242406c2a0 con 0x7f24240b2bb0 2024-04-24T23:50:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.679+0000 7f2423fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f24240cfcf0 0x7f2424161bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57628/0 (socket says 172.21.15.16:57628) 2024-04-24T23:50:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.679+0000 7f2423fff640 1 -- 172.21.15.16:0/1371578532 learned_addr learned my addr 172.21.15.16:0/1371578532 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f2422ffd640 1 --2- 172.21.15.16:0/1371578532 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24240b2bb0 0x7f24240cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.685 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24237fe640 1 --2- 172.21.15.16:0/1371578532 >> v2:172.21.15.156:3300/0 conn(0x7f242415e020 0x7f24240cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 -- 172.21.15.16:0/1371578532 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 49202570 0 0) 0x7f24240b3390 con 0x7f24240cfcf0 2024-04-24T23:50:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 -- 172.21.15.16:0/1371578532 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f240c003520 con 0x7f24240cfcf0 2024-04-24T23:50:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 -- 172.21.15.16:0/1371578532 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2358742062 0 0) 0x7f240c003520 con 0x7f24240cfcf0 2024-04-24T23:50:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 -- 172.21.15.16:0/1371578532 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24240b2bb0 msgr2=0x7f24240cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 --2- 172.21.15.16:0/1371578532 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24240b2bb0 0x7f24240cf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 -- 172.21.15.16:0/1371578532 >> v2:172.21.15.156:3300/0 conn(0x7f242415e020 msgr2=0x7f24240cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 --2- 172.21.15.16:0/1371578532 >> v2:172.21.15.156:3300/0 conn(0x7f242415e020 0x7f24240cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 -- 172.21.15.16:0/1371578532 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242406c580 con 0x7f24240cfcf0 2024-04-24T23:50:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 -- 172.21.15.16:0/1371578532 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f241c0019f0 con 0x7f24240cfcf0 2024-04-24T23:50:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 -- 172.21.15.16:0/1371578532 >> v1:172.21.15.16:6789/0 conn(0x7f24240cfcf0 legacy=0x7f2424161bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 --2- 172.21.15.16:0/1371578532 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24240b2bb0 0x7f240c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 --2- 172.21.15.16:0/1371578532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c003f30 0x7f240c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.686 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 --2- 172.21.15.16:0/1371578532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f240c004d40 0x7f240c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 -- 172.21.15.16:0/1371578532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f242406bfc0 con 0x7f240c003f30 2024-04-24T23:50:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 -- 172.21.15.16:0/1371578532 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f242406c2a0 con 0x7f240c004d40 2024-04-24T23:50:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24227fc640 1 -- 172.21.15.16:0/1371578532 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f240c00ba10 con 0x7f24240b2bb0 2024-04-24T23:50:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f2422ffd640 1 --2- 172.21.15.16:0/1371578532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c003f30 0x7f240c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f24237fe640 1 --2- 172.21.15.16:0/1371578532 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24240b2bb0 0x7f240c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f2423fff640 1 --2- 172.21.15.16:0/1371578532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f240c004d40 0x7f240c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f2422ffd640 1 -- 172.21.15.16:0/1371578532 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24240b2bb0 msgr2=0x7f240c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f2422ffd640 1 --2- 172.21.15.16:0/1371578532 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24240b2bb0 0x7f240c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f2422ffd640 1 -- 172.21.15.16:0/1371578532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f240c004d40 msgr2=0x7f240c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f2422ffd640 1 --2- 172.21.15.16:0/1371578532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f240c004d40 0x7f240c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.687 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.680+0000 7f2422ffd640 1 -- 172.21.15.16:0/1371578532 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f240c003520 con 0x7f240c003f30 2024-04-24T23:50:51.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.681+0000 7f2422ffd640 1 --2- 172.21.15.16:0/1371578532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c003f30 0x7f240c004800 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2418009af0 tx=0x7f240c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=866d6b62d996c767 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:51.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.681+0000 7f24227fc640 1 -- 172.21.15.16:0/1371578532 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418036070 con 0x7f240c003f30 2024-04-24T23:50:51.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.681+0000 7f24227fc640 1 -- 172.21.15.16:0/1371578532 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2418031400 con 0x7f240c003f30 2024-04-24T23:50:51.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.681+0000 7f24227fc640 1 -- 172.21.15.16:0/1371578532 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2418009690 con 0x7f240c003f30 2024-04-24T23:50:51.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.682+0000 7f2429eae640 1 -- 172.21.15.16:0/1371578532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c003f30 msgr2=0x7f240c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.682+0000 7f2429eae640 1 --2- 172.21.15.16:0/1371578532 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c003f30 0x7f240c004800 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2418009af0 tx=0x7f240c005830 comp rx=0 tx=0).stop 2024-04-24T23:50:51.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.682+0000 7f2423fff640 1 -- 172.21.15.16:0/1371578532 reap_dead start 2024-04-24T23:50:51.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.682+0000 7f2429eae640 1 -- 172.21.15.16:0/1371578532 shutdown_connections 2024-04-24T23:50:51.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.682+0000 7f2429eae640 1 -- 172.21.15.16:0/1371578532 >> 172.21.15.16:0/1371578532 conn(0x7f24240d3e10 msgr2=0x7f24240d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:51.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.682+0000 7f2429eae640 1 -- 172.21.15.16:0/1371578532 shutdown_connections 2024-04-24T23:50:51.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.682+0000 7f2429eae640 1 -- 172.21.15.16:0/1371578532 wait complete. 2024-04-24T23:50:51.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.683+0000 7f2429eae640 1 Processor -- start 2024-04-24T23:50:51.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.683+0000 7f2429eae640 1 -- start start 2024-04-24T23:50:51.688 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.684+0000 7f2429eae640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c003f30 0x7f242414b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.684+0000 7f2429eae640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24241f8800 0x7f24240c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.684+0000 7f2429eae640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24240c9f40 0x7f24241f4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.684+0000 7f2429eae640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f242415e400 con 0x7f240c003f30 2024-04-24T23:50:51.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.684+0000 7f2429eae640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f24240d1a80 con 0x7f24240c9f40 2024-04-24T23:50:51.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.684+0000 7f2429eae640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f24240d2fb0 con 0x7f24241f8800 2024-04-24T23:50:51.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.684+0000 7f24237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c003f30 0x7f242414b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.684+0000 7f24237fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c003f30 0x7f242414b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48994/0 (socket says 172.21.15.16:48994) 2024-04-24T23:50:51.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.684+0000 7f24237fe640 1 -- 172.21.15.16:0/2740172272 learned_addr learned my addr 172.21.15.16:0/2740172272 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:51.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.684+0000 7f2422ffd640 1 --2- 172.21.15.16:0/2740172272 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24241f8800 0x7f24240c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.684+0000 7f2423fff640 1 --2- 172.21.15.16:0/2740172272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24240c9f40 0x7f24241f4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.684+0000 7f24237fe640 1 -- 172.21.15.16:0/2740172272 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24241f8800 msgr2=0x7f24240c9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.685+0000 7f24237fe640 1 --2- 172.21.15.16:0/2740172272 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24241f8800 0x7f24240c9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.689 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.685+0000 7f24237fe640 1 -- 172.21.15.16:0/2740172272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24240c9f40 msgr2=0x7f24241f4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.685+0000 7f24237fe640 1 --2- 172.21.15.16:0/2740172272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24240c9f40 0x7f24241f4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.685+0000 7f24237fe640 1 -- 172.21.15.16:0/2740172272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24240c9250 con 0x7f240c003f30 2024-04-24T23:50:51.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.685+0000 7f24237fe640 1 --2- 172.21.15.16:0/2740172272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c003f30 0x7f242414b290 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2414011a70 tx=0x7f2414011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:51.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.685+0000 7f240bfff640 1 -- 172.21.15.16:0/2740172272 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241400fa90 con 0x7f240c003f30 2024-04-24T23:50:51.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.685+0000 7f240bfff640 1 -- 172.21.15.16:0/2740172272 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f241400fc30 con 0x7f240c003f30 2024-04-24T23:50:51.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.685+0000 7f240bfff640 1 -- 172.21.15.16:0/2740172272 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414014440 con 0x7f240c003f30 2024-04-24T23:50:51.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.685+0000 7f2429eae640 1 -- 172.21.15.16:0/2740172272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24240c81b0 con 0x7f240c003f30 2024-04-24T23:50:51.690 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.685+0000 7f2429eae640 1 -- 172.21.15.16:0/2740172272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24240c8770 con 0x7f240c003f30 2024-04-24T23:50:51.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.686+0000 7f2429eae640 1 -- 172.21.15.16:0/2740172272 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f242406dfe0 con 0x7f240c003f30 2024-04-24T23:50:51.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.686+0000 7f240bfff640 1 -- 172.21.15.16:0/2740172272 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2414012040 con 0x7f240c003f30 2024-04-24T23:50:51.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.687+0000 7f240bfff640 1 --2- 172.21.15.16:0/2740172272 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f241006be60 0x7f241006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.687+0000 7f240bfff640 1 -- 172.21.15.16:0/2740172272 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f241408b220 con 0x7f240c003f30 2024-04-24T23:50:51.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.687+0000 7f2422ffd640 1 --2- 172.21.15.16:0/2740172272 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f241006be60 0x7f241006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:51.691 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.688+0000 7f2422ffd640 1 --2- 172.21.15.16:0/2740172272 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f241006be60 0x7f241006e320 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2418033000 tx=0x7f2418008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:51.692 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.691+0000 7f240bfff640 1 -- 172.21.15.16:0/2740172272 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f2414019030 con 0x7f240c003f30 2024-04-24T23:50:51.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.876+0000 7f2429eae640 1 -- 172.21.15.16:0/2740172272 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f24240ca320 con 0x7f241006be60 2024-04-24T23:50:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.876+0000 7f240bfff640 1 -- 172.21.15.16:0/2740172272 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f24240ca320 con 0x7f241006be60 2024-04-24T23:50:51.876 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 2 to deep-scrub 2024-04-24T23:50:51.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.878+0000 7f2429eae640 1 -- 172.21.15.16:0/2740172272 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f241006be60 msgr2=0x7f241006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.878+0000 7f2429eae640 1 --2- 172.21.15.16:0/2740172272 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f241006be60 0x7f241006e320 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2418033000 tx=0x7f2418008030 comp rx=0 tx=0).stop 2024-04-24T23:50:51.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.878+0000 7f2429eae640 1 -- 172.21.15.16:0/2740172272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c003f30 msgr2=0x7f242414b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:51.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.878+0000 7f2429eae640 1 --2- 172.21.15.16:0/2740172272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c003f30 0x7f242414b290 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f2414011a70 tx=0x7f2414011e70 comp rx=0 tx=0).stop 2024-04-24T23:50:51.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.878+0000 7f2429eae640 1 -- 172.21.15.16:0/2740172272 shutdown_connections 2024-04-24T23:50:51.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.878+0000 7f2429eae640 1 --2- 172.21.15.16:0/2740172272 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f241006be60 0x7f241006e320 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.878+0000 7f2429eae640 1 --2- 172.21.15.16:0/2740172272 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f24240c9f40 0x7f24241f4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.878+0000 7f2429eae640 1 --2- 172.21.15.16:0/2740172272 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f24241f8800 0x7f24240c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.878+0000 7f2429eae640 1 --2- 172.21.15.16:0/2740172272 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f240c003f30 0x7f242414b290 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:51.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.878+0000 7f2429eae640 1 -- 172.21.15.16:0/2740172272 >> 172.21.15.16:0/2740172272 conn(0x7f24240d3e10 msgr2=0x7f2424153da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:51.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.878+0000 7f2429eae640 1 -- 172.21.15.16:0/2740172272 shutdown_connections 2024-04-24T23:50:51.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.878+0000 7f2429eae640 1 -- 172.21.15.16:0/2740172272 wait complete. 2024-04-24T23:50:51.889 INFO:tasks.ceph:Scrubbing osd.3 2024-04-24T23:50:51.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:50:51.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.994+0000 7f897e363640 1 Processor -- start 2024-04-24T23:50:51.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.994+0000 7f897e363640 1 -- start start 2024-04-24T23:50:51.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.995+0000 7f897e363640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f897815e2b0 0x7f89780711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:51.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.995+0000 7f897e363640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89780cc670 con 0x7f89781627a0 2024-04-24T23:50:51.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.995+0000 7f897e363640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89780cdad0 con 0x7f89780b4a50 2024-04-24T23:50:51.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.995+0000 7f897e363640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f897815e690 con 0x7f897815e2b0 2024-04-24T23:50:51.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.995+0000 7f89777fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f89781627a0 0x7f89780cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57636/0 (socket says 172.21.15.16:57636) 2024-04-24T23:50:51.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.995+0000 7f89777fe640 1 -- 172.21.15.16:0/2726933645 learned_addr learned my addr 172.21.15.16:0/2726933645 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:51.999 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.995+0000 7f8977fff640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f897815e2b0 0x7f89780711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.995+0000 7f8976ffd640 1 -- 172.21.15.16:0/2726933645 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3969977217 0 0) 0x7f89780cc670 con 0x7f89781627a0 2024-04-24T23:50:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.995+0000 7f8976ffd640 1 -- 172.21.15.16:0/2726933645 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8964003520 con 0x7f89781627a0 2024-04-24T23:50:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.995+0000 7f8976ffd640 1 -- 172.21.15.16:0/2726933645 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1639803233 0 0) 0x7f8964003520 con 0x7f89781627a0 2024-04-24T23:50:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.995+0000 7f8976ffd640 1 -- 172.21.15.16:0/2726933645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f897815e2b0 msgr2=0x7f89780711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.995+0000 7f8976ffd640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f897815e2b0 0x7f89780711b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.995+0000 7f8976ffd640 1 -- 172.21.15.16:0/2726933645 >> v1:172.21.15.156:6789/0 conn(0x7f89780b4a50 legacy=0x7f89780cfac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8976ffd640 1 -- 172.21.15.16:0/2726933645 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89780cc8e0 con 0x7f89781627a0 2024-04-24T23:50:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8976ffd640 1 -- 172.21.15.16:0/2726933645 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f8968001670 con 0x7f89781627a0 2024-04-24T23:50:52.000 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8977fff640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f897815e2b0 0x7f89780711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T23:50:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8976ffd640 1 -- 172.21.15.16:0/2726933645 >> v1:172.21.15.16:6789/0 conn(0x7f89781627a0 legacy=0x7f89780cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8976ffd640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89640039e0 0x7f8964003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8976ffd640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f897815e2b0 0x7f8964004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8976ffd640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8964004d40 0x7f89640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8976ffd640 1 -- 172.21.15.16:0/2726933645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f897815e690 con 0x7f89640039e0 2024-04-24T23:50:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8976ffd640 1 -- 172.21.15.16:0/2726933645 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f896400b9c0 con 0x7f8964004d40 2024-04-24T23:50:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8976ffd640 1 -- 172.21.15.16:0/2726933645 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f896400bb70 con 0x7f897815e2b0 2024-04-24T23:50:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8977fff640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89640039e0 0x7f8964003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f89777fe640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f897815e2b0 0x7f8964004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f897c900640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8964004d40 0x7f89640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.001 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8977fff640 1 -- 172.21.15.16:0/2726933645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f897815e2b0 msgr2=0x7f8964004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8977fff640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f897815e2b0 0x7f8964004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8977fff640 1 -- 172.21.15.16:0/2726933645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8964004d40 msgr2=0x7f89640051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8977fff640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8964004d40 0x7f89640051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f8977fff640 1 -- 172.21.15.16:0/2726933645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8964003520 con 0x7f89640039e0 2024-04-24T23:50:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f89777fe640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f897815e2b0 0x7f8964004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.996+0000 7f897c900640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8964004d40 0x7f89640051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.997+0000 7f8977fff640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89640039e0 0x7f8964003dc0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8960009c90 tx=0x7f8964005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd062f318df3ada9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.997+0000 7f8976ffd640 1 -- 172.21.15.16:0/2726933645 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960036070 con 0x7f89640039e0 2024-04-24T23:50:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.997+0000 7f8976ffd640 1 -- 172.21.15.16:0/2726933645 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8960031400 con 0x7f89640039e0 2024-04-24T23:50:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.997+0000 7f8976ffd640 1 -- 172.21.15.16:0/2726933645 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960009860 con 0x7f89640039e0 2024-04-24T23:50:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.997+0000 7f897e363640 1 -- 172.21.15.16:0/2726933645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89640039e0 msgr2=0x7f8964003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.997+0000 7f897e363640 1 --2- 172.21.15.16:0/2726933645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89640039e0 0x7f8964003dc0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8960009c90 tx=0x7f8964005830 comp rx=0 tx=0).stop 2024-04-24T23:50:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.997+0000 7f897c900640 1 -- 172.21.15.16:0/2726933645 reap_dead start 2024-04-24T23:50:52.002 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.997+0000 7f897e363640 1 -- 172.21.15.16:0/2726933645 shutdown_connections 2024-04-24T23:50:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.997+0000 7f897e363640 1 -- 172.21.15.16:0/2726933645 >> 172.21.15.16:0/2726933645 conn(0x7f89780d1bc0 msgr2=0x7f89780d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.997+0000 7f897e363640 1 -- 172.21.15.16:0/2726933645 shutdown_connections 2024-04-24T23:50:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.997+0000 7f897e363640 1 -- 172.21.15.16:0/2726933645 wait complete. 2024-04-24T23:50:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.999+0000 7f897e363640 1 Processor -- start 2024-04-24T23:50:52.003 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.999+0000 7f897e363640 1 -- start start 2024-04-24T23:50:52.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.999+0000 7f897e363640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89640039e0 0x7f89780ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.999+0000 7f897e363640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f897814d3e0 0x7f8978070d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.999+0000 7f897e363640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f897814c790 0x7f897814cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.999+0000 7f897e363640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8978162c30 con 0x7f89640039e0 2024-04-24T23:50:52.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.999+0000 7f897e363640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f89780ccf30 con 0x7f897814d3e0 2024-04-24T23:50:52.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:51.999+0000 7f897e363640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f89780cdf30 con 0x7f897814c790 2024-04-24T23:50:52.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f8977fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89640039e0 0x7f89780ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f8977fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89640039e0 0x7f89780ae060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49010/0 (socket says 172.21.15.16:49010) 2024-04-24T23:50:52.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f8977fff640 1 -- 172.21.15.16:0/1420428202 learned_addr learned my addr 172.21.15.16:0/1420428202 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:52.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f89777fe640 1 --2- 172.21.15.16:0/1420428202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f897814d3e0 0x7f8978070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.006 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f897c900640 1 --2- 172.21.15.16:0/1420428202 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f897814c790 0x7f897814cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f8977fff640 1 -- 172.21.15.16:0/1420428202 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f897814c790 msgr2=0x7f897814cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f8977fff640 1 --2- 172.21.15.16:0/1420428202 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f897814c790 0x7f897814cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f8977fff640 1 -- 172.21.15.16:0/1420428202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f897814d3e0 msgr2=0x7f8978070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f8977fff640 1 --2- 172.21.15.16:0/1420428202 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f897814d3e0 0x7f8978070d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f8977fff640 1 -- 172.21.15.16:0/1420428202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89780ae920 con 0x7f89640039e0 2024-04-24T23:50:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f8977fff640 1 --2- 172.21.15.16:0/1420428202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89640039e0 0x7f89780ae060 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f896002ad30 tx=0x7f8960004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f8974ff9640 1 -- 172.21.15.16:0/1420428202 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960041020 con 0x7f89640039e0 2024-04-24T23:50:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f8974ff9640 1 -- 172.21.15.16:0/1420428202 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89600093e0 con 0x7f89640039e0 2024-04-24T23:50:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f897e363640 1 -- 172.21.15.16:0/1420428202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89780aec00 con 0x7f89640039e0 2024-04-24T23:50:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f8974ff9640 1 -- 172.21.15.16:0/1420428202 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960036040 con 0x7f89640039e0 2024-04-24T23:50:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.000+0000 7f897e363640 1 -- 172.21.15.16:0/1420428202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89780af1c0 con 0x7f89640039e0 2024-04-24T23:50:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.001+0000 7f897e363640 1 -- 172.21.15.16:0/1420428202 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f89780b4660 con 0x7f89640039e0 2024-04-24T23:50:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.002+0000 7f8974ff9640 1 -- 172.21.15.16:0/1420428202 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8960044020 con 0x7f89640039e0 2024-04-24T23:50:52.007 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.002+0000 7f8974ff9640 1 --2- 172.21.15.16:0/1420428202 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f895806be30 0x7f895806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.002+0000 7f8974ff9640 1 -- 172.21.15.16:0/1420428202 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f89600ab430 con 0x7f89640039e0 2024-04-24T23:50:52.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.002+0000 7f8974ff9640 1 --2- 172.21.15.16:0/1420428202 >> [v2:172.21.15.16:6816/3747244008,v1:172.21.15.16:6817/3747244008] conn(0x7f8958072370 0x7f89580747b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.002+0000 7f8974ff9640 1 -- 172.21.15.16:0/1420428202 --> [v2:172.21.15.16:6816/3747244008,v1:172.21.15.16:6817/3747244008] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8960033e90 con 0x7f8958072370 2024-04-24T23:50:52.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.002+0000 7f8974ff9640 1 -- 172.21.15.16:0/1420428202 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f89600b2050 con 0x7f89640039e0 2024-04-24T23:50:52.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.002+0000 7f89777fe640 1 --2- 172.21.15.16:0/1420428202 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f895806be30 0x7f895806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.002+0000 7f897c900640 1 --2- 172.21.15.16:0/1420428202 >> [v2:172.21.15.16:6816/3747244008,v1:172.21.15.16:6817/3747244008] conn(0x7f8958072370 0x7f89580747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.003+0000 7f897c900640 1 --2- 172.21.15.16:0/1420428202 >> [v2:172.21.15.16:6816/3747244008,v1:172.21.15.16:6817/3747244008] conn(0x7f8958072370 0x7f89580747b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.003+0000 7f89777fe640 1 --2- 172.21.15.16:0/1420428202 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f895806be30 0x7f895806e2f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f89780cbe60 tx=0x7f8968007040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.008 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.003+0000 7f8974ff9640 1 -- 172.21.15.16:0/1420428202 <== osd.3 v2:172.21.15.16:6816/3747244008 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8960033e90 con 0x7f8958072370 2024-04-24T23:50:52.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.017+0000 7f897e363640 1 -- 172.21.15.16:0/1420428202 --> [v2:172.21.15.16:6816/3747244008,v1:172.21.15.16:6817/3747244008] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8978068f90 con 0x7f8958072370 2024-04-24T23:50:52.016 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.017+0000 7f8974ff9640 1 -- 172.21.15.16:0/1420428202 <== osd.3 v2:172.21.15.16:6816/3747244008 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8978068f90 con 0x7f8958072370 2024-04-24T23:50:52.017 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:50:52.017 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:50:52.017 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:50:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.017+0000 7f897e363640 1 -- 172.21.15.16:0/1420428202 >> [v2:172.21.15.16:6816/3747244008,v1:172.21.15.16:6817/3747244008] conn(0x7f8958072370 msgr2=0x7f89580747b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.017+0000 7f897e363640 1 --2- 172.21.15.16:0/1420428202 >> [v2:172.21.15.16:6816/3747244008,v1:172.21.15.16:6817/3747244008] conn(0x7f8958072370 0x7f89580747b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.017+0000 7f897e363640 1 -- 172.21.15.16:0/1420428202 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f895806be30 msgr2=0x7f895806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.017+0000 7f897e363640 1 --2- 172.21.15.16:0/1420428202 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f895806be30 0x7f895806e2f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f89780cbe60 tx=0x7f8968007040 comp rx=0 tx=0).stop 2024-04-24T23:50:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.017+0000 7f897e363640 1 -- 172.21.15.16:0/1420428202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89640039e0 msgr2=0x7f89780ae060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.017+0000 7f897e363640 1 --2- 172.21.15.16:0/1420428202 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f89640039e0 0x7f89780ae060 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f896002ad30 tx=0x7f8960004730 comp rx=0 tx=0).stop 2024-04-24T23:50:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.017+0000 7f897c900640 1 -- 172.21.15.16:0/1420428202 reap_dead start 2024-04-24T23:50:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.017+0000 7f897e363640 1 -- 172.21.15.16:0/1420428202 shutdown_connections 2024-04-24T23:50:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.017+0000 7f897e363640 1 -- 172.21.15.16:0/1420428202 >> 172.21.15.16:0/1420428202 conn(0x7f89780d1bc0 msgr2=0x7f89781e9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.017+0000 7f897e363640 1 -- 172.21.15.16:0/1420428202 shutdown_connections 2024-04-24T23:50:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.017+0000 7f897e363640 1 -- 172.21.15.16:0/1420428202 wait complete. 2024-04-24T23:50:52.027 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-04-24T23:50:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.129+0000 7f8a29c66640 1 Processor -- start 2024-04-24T23:50:52.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.129+0000 7f8a29c66640 1 -- start start 2024-04-24T23:50:52.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.130+0000 7f8a29c66640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8a240d2fb0 0x7f8a24168f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.130+0000 7f8a29c66640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a240d0090 con 0x7f8a240b2bb0 2024-04-24T23:50:52.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.130+0000 7f8a29c66640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a240b3390 con 0x7f8a2415e020 2024-04-24T23:50:52.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.130+0000 7f8a29c66640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8a2406bfc0 con 0x7f8a240d2fb0 2024-04-24T23:50:52.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.130+0000 7f8a22ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8a240b2bb0 0x7f8a240cf980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57638/0 (socket says 172.21.15.16:57638) 2024-04-24T23:50:52.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.130+0000 7f8a22ffd640 1 -- 172.21.15.16:0/1563771262 learned_addr learned my addr 172.21.15.16:0/1563771262 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:52.133 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a23fff640 1 --2- 172.21.15.16:0/1563771262 >> v2:172.21.15.156:3300/0 conn(0x7f8a240d2fb0 0x7f8a24168f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4234784831 0 0) 0x7f8a240d0090 con 0x7f8a240b2bb0 2024-04-24T23:50:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a0c003520 con 0x7f8a240b2bb0 2024-04-24T23:50:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 <== mon.1 v1:172.21.15.156:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 568069588 0 0) 0x7f8a240b3390 con 0x7f8a2415e020 2024-04-24T23:50:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 --> v1:172.21.15.156:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a240d0090 con 0x7f8a2415e020 2024-04-24T23:50:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1822817104 0 0) 0x7f8a0c003520 con 0x7f8a240b2bb0 2024-04-24T23:50:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 >> v2:172.21.15.156:3300/0 conn(0x7f8a240d2fb0 msgr2=0x7f8a24168f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 --2- 172.21.15.16:0/1563771262 >> v2:172.21.15.156:3300/0 conn(0x7f8a240d2fb0 0x7f8a24168f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 >> v1:172.21.15.156:6789/0 conn(0x7f8a2415e020 legacy=0x7f8a240cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2406c2a0 con 0x7f8a240b2bb0 2024-04-24T23:50:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f8a18001670 con 0x7f8a240b2bb0 2024-04-24T23:50:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 >> v1:172.21.15.16:6789/0 conn(0x7f8a240b2bb0 legacy=0x7f8a240cf980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 --2- 172.21.15.16:0/1563771262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a0c003b40 0x7f8a0c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 --2- 172.21.15.16:0/1563771262 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a0c004800 0x7f8a0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 --2- 172.21.15.16:0/1563771262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c0051c0 0x7f8a0c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2406bfc0 con 0x7f8a0c0051c0 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0c00c0f0 con 0x7f8a0c003b40 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8a0c00c2f0 con 0x7f8a0c004800 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a23fff640 1 --2- 172.21.15.16:0/1563771262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c0051c0 0x7f8a0c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a22ffd640 1 --2- 172.21.15.16:0/1563771262 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a0c004800 0x7f8a0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a237fe640 1 --2- 172.21.15.16:0/1563771262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a0c003b40 0x7f8a0c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a23fff640 1 -- 172.21.15.16:0/1563771262 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a0c004800 msgr2=0x7f8a0c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a23fff640 1 --2- 172.21.15.16:0/1563771262 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a0c004800 0x7f8a0c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a23fff640 1 -- 172.21.15.16:0/1563771262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a0c003b40 msgr2=0x7f8a0c003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a23fff640 1 --2- 172.21.15.16:0/1563771262 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a0c003b40 0x7f8a0c003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.131+0000 7f8a23fff640 1 -- 172.21.15.16:0/1563771262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0c003520 con 0x7f8a0c0051c0 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.132+0000 7f8a23fff640 1 -- 172.21.15.16:0/1563771262 reap_dead start 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.132+0000 7f8a23fff640 1 --2- 172.21.15.16:0/1563771262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c0051c0 0x7f8a0c00b9c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009fc0 tx=0x7f8a0c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a1e433329243ad6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.132+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c035020 con 0x7f8a0c0051c0 2024-04-24T23:50:52.135 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.132+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a1c003bd0 con 0x7f8a0c0051c0 2024-04-24T23:50:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.132+0000 7f8a227fc640 1 -- 172.21.15.16:0/1563771262 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c009c10 con 0x7f8a0c0051c0 2024-04-24T23:50:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.133+0000 7f8a29c66640 1 -- 172.21.15.16:0/1563771262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c0051c0 msgr2=0x7f8a0c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.133+0000 7f8a29c66640 1 --2- 172.21.15.16:0/1563771262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c0051c0 0x7f8a0c00b9c0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c009fc0 tx=0x7f8a0c005830 comp rx=0 tx=0).stop 2024-04-24T23:50:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.133+0000 7f8a29c66640 1 -- 172.21.15.16:0/1563771262 shutdown_connections 2024-04-24T23:50:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.133+0000 7f8a29c66640 1 --2- 172.21.15.16:0/1563771262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a0c0051c0 0x7f8a0c00b9c0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.133+0000 7f8a29c66640 1 -- 172.21.15.16:0/1563771262 >> 172.21.15.16:0/1563771262 conn(0x7f8a240d3e10 msgr2=0x7f8a240d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.133+0000 7f8a29c66640 1 -- 172.21.15.16:0/1563771262 shutdown_connections 2024-04-24T23:50:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.133+0000 7f8a29c66640 1 -- 172.21.15.16:0/1563771262 wait complete. 2024-04-24T23:50:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.134+0000 7f8a29c66640 1 Processor -- start 2024-04-24T23:50:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.134+0000 7f8a29c66640 1 -- start start 2024-04-24T23:50:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a29c66640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a0c0051c0 0x7f8a24157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a29c66640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a240b9690 0x7f8a2414d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a29c66640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a240cc620 0x7f8a24158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a29c66640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8a240d1a80 con 0x7f8a240cc620 2024-04-24T23:50:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a29c66640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2406c860 con 0x7f8a240b9690 2024-04-24T23:50:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a29c66640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8a2415e400 con 0x7f8a0c0051c0 2024-04-24T23:50:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a23fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a240cc620 0x7f8a24158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a23fff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a240cc620 0x7f8a24158760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49026/0 (socket says 172.21.15.16:49026) 2024-04-24T23:50:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a23fff640 1 -- 172.21.15.16:0/2595820079 learned_addr learned my addr 172.21.15.16:0/2595820079 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a23fff640 1 -- 172.21.15.16:0/2595820079 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a0c0051c0 msgr2=0x7f8a24157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a23fff640 1 --2- 172.21.15.16:0/2595820079 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a0c0051c0 0x7f8a24157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a23fff640 1 -- 172.21.15.16:0/2595820079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a240b9690 msgr2=0x7f8a2414d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a23fff640 1 --2- 172.21.15.16:0/2595820079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a240b9690 0x7f8a2414d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a23fff640 1 -- 172.21.15.16:0/2595820079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a240cb8c0 con 0x7f8a240cc620 2024-04-24T23:50:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.135+0000 7f8a23fff640 1 --2- 172.21.15.16:0/2595820079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a240cc620 0x7f8a24158760 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c02af10 tx=0x7f8a1c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.136+0000 7f8a0bfff640 1 -- 172.21.15.16:0/2595820079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c040070 con 0x7f8a240cc620 2024-04-24T23:50:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.136+0000 7f8a0bfff640 1 -- 172.21.15.16:0/2595820079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a1c032e80 con 0x7f8a240cc620 2024-04-24T23:50:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.136+0000 7f8a29c66640 1 -- 172.21.15.16:0/2595820079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a2414d710 con 0x7f8a240cc620 2024-04-24T23:50:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.136+0000 7f8a0bfff640 1 -- 172.21.15.16:0/2595820079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c035030 con 0x7f8a240cc620 2024-04-24T23:50:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.136+0000 7f8a29c66640 1 -- 172.21.15.16:0/2595820079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a240cbb40 con 0x7f8a240cc620 2024-04-24T23:50:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.137+0000 7f8a29c66640 1 -- 172.21.15.16:0/2595820079 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a2406dfe0 con 0x7f8a240cc620 2024-04-24T23:50:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.137+0000 7f8a0bfff640 1 -- 172.21.15.16:0/2595820079 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8a1c043070 con 0x7f8a240cc620 2024-04-24T23:50:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.137+0000 7f8a0bfff640 1 --2- 172.21.15.16:0/2595820079 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a1006be30 0x7f8a1006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.138+0000 7f8a0bfff640 1 -- 172.21.15.16:0/2595820079 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f8a1c0abdc0 con 0x7f8a240cc620 2024-04-24T23:50:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.138+0000 7f8a237fe640 1 --2- 172.21.15.16:0/2595820079 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a1006be30 0x7f8a1006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.138+0000 7f8a237fe640 1 --2- 172.21.15.16:0/2595820079 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a1006be30 0x7f8a1006e2f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8a14007850 tx=0x7f8a14009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.142+0000 7f8a0bfff640 1 -- 172.21.15.16:0/2595820079 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8a240cb8c0 con 0x7f8a240cc620 2024-04-24T23:50:52.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.324+0000 7f8a29c66640 1 -- 172.21.15.16:0/2595820079 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f8a240ced00 con 0x7f8a1006be30 2024-04-24T23:50:52.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.325+0000 7f8a0bfff640 1 -- 172.21.15.16:0/2595820079 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8a240ced00 con 0x7f8a1006be30 2024-04-24T23:50:52.325 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 3 to deep-scrub 2024-04-24T23:50:52.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.327+0000 7f8a29c66640 1 -- 172.21.15.16:0/2595820079 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a1006be30 msgr2=0x7f8a1006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.327+0000 7f8a29c66640 1 --2- 172.21.15.16:0/2595820079 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a1006be30 0x7f8a1006e2f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8a14007850 tx=0x7f8a14009040 comp rx=0 tx=0).stop 2024-04-24T23:50:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.327+0000 7f8a29c66640 1 -- 172.21.15.16:0/2595820079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a240cc620 msgr2=0x7f8a24158760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.327+0000 7f8a29c66640 1 --2- 172.21.15.16:0/2595820079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a240cc620 0x7f8a24158760 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c02af10 tx=0x7f8a1c009750 comp rx=0 tx=0).stop 2024-04-24T23:50:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.327+0000 7f8a29c66640 1 -- 172.21.15.16:0/2595820079 shutdown_connections 2024-04-24T23:50:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.327+0000 7f8a29c66640 1 --2- 172.21.15.16:0/2595820079 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8a1006be30 0x7f8a1006e2f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.327+0000 7f8a29c66640 1 --2- 172.21.15.16:0/2595820079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8a240cc620 0x7f8a24158760 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.327+0000 7f8a29c66640 1 --2- 172.21.15.16:0/2595820079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8a240b9690 0x7f8a2414d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.327+0000 7f8a29c66640 1 --2- 172.21.15.16:0/2595820079 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8a0c0051c0 0x7f8a24157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.327+0000 7f8a29c66640 1 -- 172.21.15.16:0/2595820079 >> 172.21.15.16:0/2595820079 conn(0x7f8a240d3e10 msgr2=0x7f8a240b7430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.327+0000 7f8a29c66640 1 -- 172.21.15.16:0/2595820079 shutdown_connections 2024-04-24T23:50:52.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.327+0000 7f8a29c66640 1 -- 172.21.15.16:0/2595820079 wait complete. 2024-04-24T23:50:52.338 INFO:tasks.ceph:Scrubbing osd.4 2024-04-24T23:50:52.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:50:52.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.442+0000 7f25641e7640 1 Processor -- start 2024-04-24T23:50:52.442 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.442+0000 7f25641e7640 1 -- start start 2024-04-24T23:50:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.443+0000 7f25641e7640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f255c06c110 0x7f255c0d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.443+0000 7f25641e7640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f255c0d1aa0 con 0x7f255c15dfb0 2024-04-24T23:50:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.443+0000 7f25641e7640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f255c06b470 con 0x7f255c06bcb0 2024-04-24T23:50:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.443+0000 7f25641e7640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f255c0d02b0 con 0x7f255c06c110 2024-04-24T23:50:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.443+0000 7f2562784640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f255c06c110 0x7f255c0d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.443 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.443+0000 7f2561f83640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f255c15dfb0 0x7f255c0cfba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57654/0 (socket says 172.21.15.16:57654) 2024-04-24T23:50:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.443+0000 7f2561f83640 1 -- 172.21.15.16:0/3218263246 learned_addr learned my addr 172.21.15.16:0/3218263246 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.444+0000 7f2562784640 1 -- 172.21.15.16:0/3218263246 >> v1:172.21.15.156:6789/0 conn(0x7f255c06bcb0 legacy=0x7f255c0d2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.444+0000 7f2562784640 1 -- 172.21.15.16:0/3218263246 >> v1:172.21.15.16:6789/0 conn(0x7f255c15dfb0 legacy=0x7f255c0cfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.444+0000 7f2562784640 1 -- 172.21.15.16:0/3218263246 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f255c06b730 con 0x7f255c06c110 2024-04-24T23:50:52.444 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.444+0000 7f2560f81640 1 -- 172.21.15.16:0/3218263246 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4243554503 0 0) 0x7f255c0d1aa0 con 0x7f255c15dfb0 2024-04-24T23:50:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.444+0000 7f2562784640 1 --2- 172.21.15.16:0/3218263246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f255c06c110 0x7f255c0d2970 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2554009d90 tx=0x7f255402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a2e85a57fd700e8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.444+0000 7f2560f81640 1 -- 172.21.15.16:0/3218263246 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2554007740 con 0x7f255c06c110 2024-04-24T23:50:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.445+0000 7f2560f81640 1 -- 172.21.15.16:0/3218263246 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25540078e0 con 0x7f255c06c110 2024-04-24T23:50:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.445+0000 7f2560f81640 1 -- 172.21.15.16:0/3218263246 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2554007cb0 con 0x7f255c06c110 2024-04-24T23:50:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.445+0000 7f25641e7640 1 -- 172.21.15.16:0/3218263246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f255c06c110 msgr2=0x7f255c0d2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.445+0000 7f25641e7640 1 --2- 172.21.15.16:0/3218263246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f255c06c110 0x7f255c0d2970 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f2554009d90 tx=0x7f255402a6a0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.445+0000 7f25641e7640 1 -- 172.21.15.16:0/3218263246 shutdown_connections 2024-04-24T23:50:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.445+0000 7f25641e7640 1 --2- 172.21.15.16:0/3218263246 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f255c06c110 0x7f255c0d2970 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.445 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.445+0000 7f25641e7640 1 -- 172.21.15.16:0/3218263246 >> 172.21.15.16:0/3218263246 conn(0x7f255c0d3e30 msgr2=0x7f255c0d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.445+0000 7f25641e7640 1 -- 172.21.15.16:0/3218263246 shutdown_connections 2024-04-24T23:50:52.446 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.445+0000 7f25641e7640 1 -- 172.21.15.16:0/3218263246 wait complete. 2024-04-24T23:50:52.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.446+0000 7f25641e7640 1 Processor -- start 2024-04-24T23:50:52.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.446+0000 7f25641e7640 1 -- start start 2024-04-24T23:50:52.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.447+0000 7f25641e7640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f255c06bcb0 0x7f255c14b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.447+0000 7f25641e7640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f255c06c110 0x7f255c15f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.447+0000 7f25641e7640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f255c15dfb0 0x7f255c1f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.447+0000 7f25641e7640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f255c06ba10 con 0x7f255c15dfb0 2024-04-24T23:50:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.447+0000 7f25641e7640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f255c0d1dd0 con 0x7f255c06c110 2024-04-24T23:50:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.447+0000 7f25641e7640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f255c06c560 con 0x7f255c06bcb0 2024-04-24T23:50:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.447+0000 7f2562784640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f255c15dfb0 0x7f255c1f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.447+0000 7f2562784640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f255c15dfb0 0x7f255c1f3e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49052/0 (socket says 172.21.15.16:49052) 2024-04-24T23:50:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.447+0000 7f2562784640 1 -- 172.21.15.16:0/743480878 learned_addr learned my addr 172.21.15.16:0/743480878 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f2561782640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f255c06c110 0x7f255c15f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f2561f83640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f255c06bcb0 0x7f255c14b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f2562784640 1 -- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f255c06bcb0 msgr2=0x7f255c14b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f2562784640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f255c06bcb0 0x7f255c14b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f2562784640 1 -- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f255c06c110 msgr2=0x7f255c15f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f2562784640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f255c06c110 0x7f255c15f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f2562784640 1 -- 172.21.15.16:0/743480878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f255c14b650 con 0x7f255c15dfb0 2024-04-24T23:50:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f2561782640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f255c06c110 0x7f255c15f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:52.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f2561f83640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f255c06bcb0 0x7f255c14b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f2562784640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f255c15dfb0 0x7f255c1f3e60 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2554009e80 tx=0x7f2554004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f25527fc640 1 -- 172.21.15.16:0/743480878 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2554040070 con 0x7f255c15dfb0 2024-04-24T23:50:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f25527fc640 1 -- 172.21.15.16:0/743480878 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2554031cb0 con 0x7f255c15dfb0 2024-04-24T23:50:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f25641e7640 1 -- 172.21.15.16:0/743480878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f255c15fa60 con 0x7f255c15dfb0 2024-04-24T23:50:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f25527fc640 1 -- 172.21.15.16:0/743480878 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2554031e60 con 0x7f255c15dfb0 2024-04-24T23:50:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.448+0000 7f25641e7640 1 -- 172.21.15.16:0/743480878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f255c160020 con 0x7f255c15dfb0 2024-04-24T23:50:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.449+0000 7f25641e7640 1 -- 172.21.15.16:0/743480878 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f255c068f90 con 0x7f255c15dfb0 2024-04-24T23:50:52.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.449+0000 7f25527fc640 1 -- 172.21.15.16:0/743480878 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2554039040 con 0x7f255c15dfb0 2024-04-24T23:50:52.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.450+0000 7f25527fc640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f254006be30 0x7f254006e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.450+0000 7f25527fc640 1 -- 172.21.15.16:0/743480878 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f25540ac780 con 0x7f255c15dfb0 2024-04-24T23:50:52.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.450+0000 7f25527fc640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:6808/542868407,v1:172.21.15.156:6809/542868407] conn(0x7f2540072370 0x7f25400747b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.450+0000 7f25527fc640 1 -- 172.21.15.16:0/743480878 --> [v2:172.21.15.156:6808/542868407,v1:172.21.15.156:6809/542868407] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2554043e50 con 0x7f2540072370 2024-04-24T23:50:52.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.450+0000 7f25527fc640 1 -- 172.21.15.16:0/743480878 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f25540b3050 con 0x7f255c15dfb0 2024-04-24T23:50:52.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.450+0000 7f2561f83640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f254006be30 0x7f254006e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.450+0000 7f2561782640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:6808/542868407,v1:172.21.15.156:6809/542868407] conn(0x7f2540072370 0x7f25400747b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.451+0000 7f2561f83640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f254006be30 0x7f254006e2f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2558000f60 tx=0x7f2558009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.451+0000 7f2561782640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:6808/542868407,v1:172.21.15.156:6809/542868407] conn(0x7f2540072370 0x7f25400747b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.452+0000 7f25527fc640 1 -- 172.21.15.16:0/743480878 <== osd.4 v2:172.21.15.156:6808/542868407 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2554043e50 con 0x7f2540072370 2024-04-24T23:50:52.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.465+0000 7f25641e7640 1 -- 172.21.15.16:0/743480878 --> [v2:172.21.15.156:6808/542868407,v1:172.21.15.156:6809/542868407] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f255c0cecb0 con 0x7f2540072370 2024-04-24T23:50:52.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.465+0000 7f25527fc640 1 -- 172.21.15.16:0/743480878 <== osd.4 v2:172.21.15.156:6808/542868407 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f255c0cecb0 con 0x7f2540072370 2024-04-24T23:50:52.466 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:50:52.466 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:50:52.466 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:50:52.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.466+0000 7f25641e7640 1 -- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:6808/542868407,v1:172.21.15.156:6809/542868407] conn(0x7f2540072370 msgr2=0x7f25400747b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.466+0000 7f25641e7640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:6808/542868407,v1:172.21.15.156:6809/542868407] conn(0x7f2540072370 0x7f25400747b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.466+0000 7f25641e7640 1 -- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f254006be30 msgr2=0x7f254006e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.466+0000 7f25641e7640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f254006be30 0x7f254006e2f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2558000f60 tx=0x7f2558009040 comp rx=0 tx=0).stop 2024-04-24T23:50:52.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.466+0000 7f25641e7640 1 -- 172.21.15.16:0/743480878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f255c15dfb0 msgr2=0x7f255c1f3e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.466+0000 7f25641e7640 1 --2- 172.21.15.16:0/743480878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f255c15dfb0 0x7f255c1f3e60 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f2554009e80 tx=0x7f2554004a40 comp rx=0 tx=0).stop 2024-04-24T23:50:52.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.466+0000 7f2562784640 1 -- 172.21.15.16:0/743480878 reap_dead start 2024-04-24T23:50:52.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.466+0000 7f25641e7640 1 -- 172.21.15.16:0/743480878 shutdown_connections 2024-04-24T23:50:52.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.466+0000 7f25641e7640 1 -- 172.21.15.16:0/743480878 >> 172.21.15.16:0/743480878 conn(0x7f255c0d3e30 msgr2=0x7f255c0b3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:52.467 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.466+0000 7f25641e7640 1 -- 172.21.15.16:0/743480878 shutdown_connections 2024-04-24T23:50:52.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.466+0000 7f25641e7640 1 -- 172.21.15.16:0/743480878 wait complete. 2024-04-24T23:50:52.476 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-04-24T23:50:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.578+0000 7f93e897e640 1 Processor -- start 2024-04-24T23:50:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.578+0000 7f93e897e640 1 -- start start 2024-04-24T23:50:52.582 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.579+0000 7f93e897e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93e415e020 0x7f93e40cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.579+0000 7f93e897e640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f93e40b2bb0 0x7f93e40cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.579+0000 7f93e897e640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f93e40cfcf0 0x7f93e4161bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.579+0000 7f93e897e640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f93e40b3390 con 0x7f93e40cfcf0 2024-04-24T23:50:52.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.579+0000 7f93e897e640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f93e406bfc0 con 0x7f93e40b2bb0 2024-04-24T23:50:52.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.579+0000 7f93e897e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f93e406c2a0 con 0x7f93e415e020 2024-04-24T23:50:52.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.579+0000 7f93e2d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f93e40cfcf0 0x7f93e4161bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.580+0000 7f93e2d9d640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f93e40cfcf0 0x7f93e4161bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49060/0 (socket says 172.21.15.16:49060) 2024-04-24T23:50:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.580+0000 7f93e2d9d640 1 -- 172.21.15.16:0/1954928662 learned_addr learned my addr 172.21.15.16:0/1954928662 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.580+0000 7f93e259c640 1 --2- 172.21.15.16:0/1954928662 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93e415e020 0x7f93e40cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.580+0000 7f93e1d9b640 1 --2- 172.21.15.16:0/1954928662 >> v2:172.21.15.156:3300/0 conn(0x7f93e40b2bb0 0x7f93e40cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.580+0000 7f93e2d9d640 1 -- 172.21.15.16:0/1954928662 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93e415e020 msgr2=0x7f93e40cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.580+0000 7f93e2d9d640 1 --2- 172.21.15.16:0/1954928662 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93e415e020 0x7f93e40cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.580+0000 7f93e2d9d640 1 -- 172.21.15.16:0/1954928662 >> v2:172.21.15.156:3300/0 conn(0x7f93e40b2bb0 msgr2=0x7f93e40cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.580+0000 7f93e2d9d640 1 --2- 172.21.15.16:0/1954928662 >> v2:172.21.15.156:3300/0 conn(0x7f93e40b2bb0 0x7f93e40cf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.580+0000 7f93e2d9d640 1 -- 172.21.15.16:0/1954928662 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e406c580 con 0x7f93e40cfcf0 2024-04-24T23:50:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.581+0000 7f93e2d9d640 1 --2- 172.21.15.16:0/1954928662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93e40cfcf0 0x7f93e4161bf0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f93e40b6a00 tx=0x7f93d402a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2110ae9d982db73e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.581+0000 7f93e159a640 1 -- 172.21.15.16:0/1954928662 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d4036070 con 0x7f93e40cfcf0 2024-04-24T23:50:52.584 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.581+0000 7f93e159a640 1 -- 172.21.15.16:0/1954928662 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93d4031600 con 0x7f93e40cfcf0 2024-04-24T23:50:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.581+0000 7f93e159a640 1 -- 172.21.15.16:0/1954928662 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93d40319d0 con 0x7f93e40cfcf0 2024-04-24T23:50:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.581+0000 7f93e897e640 1 -- 172.21.15.16:0/1954928662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93e40cfcf0 msgr2=0x7f93e4161bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.581+0000 7f93e897e640 1 --2- 172.21.15.16:0/1954928662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93e40cfcf0 0x7f93e4161bf0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f93e40b6a00 tx=0x7f93d402a610 comp rx=0 tx=0).stop 2024-04-24T23:50:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.581+0000 7f93e897e640 1 -- 172.21.15.16:0/1954928662 shutdown_connections 2024-04-24T23:50:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.581+0000 7f93e897e640 1 --2- 172.21.15.16:0/1954928662 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93e40cfcf0 0x7f93e4161bf0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.581+0000 7f93e897e640 1 --2- 172.21.15.16:0/1954928662 >> v2:172.21.15.156:3300/0 conn(0x7f93e40b2bb0 0x7f93e40cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.581+0000 7f93e897e640 1 --2- 172.21.15.16:0/1954928662 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93e415e020 0x7f93e40cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.581+0000 7f93e897e640 1 -- 172.21.15.16:0/1954928662 >> 172.21.15.16:0/1954928662 conn(0x7f93e40d3e10 msgr2=0x7f93e40d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.581+0000 7f93e897e640 1 -- 172.21.15.16:0/1954928662 shutdown_connections 2024-04-24T23:50:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.581+0000 7f93e897e640 1 -- 172.21.15.16:0/1954928662 wait complete. 2024-04-24T23:50:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.583+0000 7f93e897e640 1 Processor -- start 2024-04-24T23:50:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.583+0000 7f93e897e640 1 -- start start 2024-04-24T23:50:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.583+0000 7f93e897e640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93e40b2bb0 0x7f93e40c8970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.583+0000 7f93e897e640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93e40cfcf0 0x7f93e4158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.583+0000 7f93e897e640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93e415e020 0x7f93e41609e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.583+0000 7f93e897e640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f93e415e400 con 0x7f93e40b2bb0 2024-04-24T23:50:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.583+0000 7f93e897e640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f93e40d1a80 con 0x7f93e40cfcf0 2024-04-24T23:50:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.583+0000 7f93e897e640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f93e4162130 con 0x7f93e415e020 2024-04-24T23:50:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93e259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93e40b2bb0 0x7f93e40c8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93e259c640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93e40b2bb0 0x7f93e40c8970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49072/0 (socket says 172.21.15.16:49072) 2024-04-24T23:50:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93e259c640 1 -- 172.21.15.16:0/2640498565 learned_addr learned my addr 172.21.15.16:0/2640498565 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:52.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93e2d9d640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93e415e020 0x7f93e41609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93e1d9b640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93e40cfcf0 0x7f93e4158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93e259c640 1 -- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93e415e020 msgr2=0x7f93e41609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93e259c640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93e415e020 0x7f93e41609e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93e259c640 1 -- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93e40cfcf0 msgr2=0x7f93e4158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93e259c640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93e40cfcf0 0x7f93e4158270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93e259c640 1 -- 172.21.15.16:0/2640498565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93e41f41b0 con 0x7f93e40b2bb0 2024-04-24T23:50:52.591 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93e1d9b640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93e40cfcf0 0x7f93e4158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:52.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93e259c640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93e40b2bb0 0x7f93e40c8970 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00e8a0 tx=0x7f93dc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93d2ffd640 1 -- 172.21.15.16:0/2640498565 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93dc017070 con 0x7f93e40b2bb0 2024-04-24T23:50:52.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93d2ffd640 1 -- 172.21.15.16:0/2640498565 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f93dc00c940 con 0x7f93e40b2bb0 2024-04-24T23:50:52.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93d2ffd640 1 -- 172.21.15.16:0/2640498565 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93dc00cd30 con 0x7f93e40b2bb0 2024-04-24T23:50:52.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93e897e640 1 -- 172.21.15.16:0/2640498565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93e4158cf0 con 0x7f93e40b2bb0 2024-04-24T23:50:52.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.584+0000 7f93e897e640 1 -- 172.21.15.16:0/2640498565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93e41601e0 con 0x7f93e40b2bb0 2024-04-24T23:50:52.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.585+0000 7f93e897e640 1 -- 172.21.15.16:0/2640498565 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f93e415fd10 con 0x7f93e40b2bb0 2024-04-24T23:50:52.592 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.586+0000 7f93d2ffd640 1 -- 172.21.15.16:0/2640498565 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f93dc01a070 con 0x7f93e40b2bb0 2024-04-24T23:50:52.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.586+0000 7f93d2ffd640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f93c406be30 0x7f93c406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.586+0000 7f93d2ffd640 1 -- 172.21.15.16:0/2640498565 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f93dc08ab50 con 0x7f93e40b2bb0 2024-04-24T23:50:52.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.586+0000 7f93e1d9b640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f93c406be30 0x7f93c406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.587+0000 7f93e1d9b640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f93c406be30 0x7f93c406e2f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f93e415cf10 tx=0x7f93d8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.593 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.590+0000 7f93d2ffd640 1 -- 172.21.15.16:0/2640498565 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f93dc058fd0 con 0x7f93e40b2bb0 2024-04-24T23:50:52.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.773+0000 7f93e897e640 1 -- 172.21.15.16:0/2640498565 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f93e406dfe0 con 0x7f93c406be30 2024-04-24T23:50:52.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.774+0000 7f93d2ffd640 1 -- 172.21.15.16:0/2640498565 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f93e406dfe0 con 0x7f93c406be30 2024-04-24T23:50:52.773 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 4 to deep-scrub 2024-04-24T23:50:52.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.775+0000 7f93e897e640 1 -- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f93c406be30 msgr2=0x7f93c406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.775+0000 7f93e897e640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f93c406be30 0x7f93c406e2f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f93e415cf10 tx=0x7f93d8008040 comp rx=0 tx=0).stop 2024-04-24T23:50:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.775+0000 7f93e897e640 1 -- 172.21.15.16:0/2640498565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93e40b2bb0 msgr2=0x7f93e40c8970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.775+0000 7f93e897e640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93e40b2bb0 0x7f93e40c8970 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f93dc00e8a0 tx=0x7f93dc00ed60 comp rx=0 tx=0).stop 2024-04-24T23:50:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.776+0000 7f93e897e640 1 -- 172.21.15.16:0/2640498565 shutdown_connections 2024-04-24T23:50:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.776+0000 7f93e897e640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f93c406be30 0x7f93c406e2f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.776+0000 7f93e897e640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f93e415e020 0x7f93e41609e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.776+0000 7f93e897e640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f93e40cfcf0 0x7f93e4158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.776+0000 7f93e897e640 1 --2- 172.21.15.16:0/2640498565 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93e40b2bb0 0x7f93e40c8970 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.776+0000 7f93e897e640 1 -- 172.21.15.16:0/2640498565 >> 172.21.15.16:0/2640498565 conn(0x7f93e40d3e10 msgr2=0x7f93e40b3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.776+0000 7f93e897e640 1 -- 172.21.15.16:0/2640498565 shutdown_connections 2024-04-24T23:50:52.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.776+0000 7f93e897e640 1 -- 172.21.15.16:0/2640498565 wait complete. 2024-04-24T23:50:52.786 INFO:tasks.ceph:Scrubbing osd.5 2024-04-24T23:50:52.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:50:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.891+0000 7f8eaef5a640 1 Processor -- start 2024-04-24T23:50:52.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.891+0000 7f8eaef5a640 1 -- start start 2024-04-24T23:50:52.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.891+0000 7f8eaef5a640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8ea8145630 0x7f8ea806e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.891 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.891+0000 7f8eaef5a640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ea806d7c0 con 0x7f8ea806e7c0 2024-04-24T23:50:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.891+0000 7f8eaef5a640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ea8160660 con 0x7f8ea80b54b0 2024-04-24T23:50:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.891+0000 7f8eaef5a640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8ea806d560 con 0x7f8ea8145630 2024-04-24T23:50:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.892+0000 7f8eaccf6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8ea8145630 0x7f8ea806e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.892+0000 7f8ea7fff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8ea806e7c0 0x7f8ea80b4da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57664/0 (socket says 172.21.15.16:57664) 2024-04-24T23:50:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.892+0000 7f8eaccf6640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8ea8145630 0x7f8ea806e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49082/0 (socket says 172.21.15.16:49082) 2024-04-24T23:50:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.892+0000 7f8ea7fff640 1 -- 172.21.15.16:0/1611967897 learned_addr learned my addr 172.21.15.16:0/1611967897 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.892+0000 7f8eaccf6640 1 -- 172.21.15.16:0/1611967897 >> v1:172.21.15.156:6789/0 conn(0x7f8ea80b54b0 legacy=0x7f8ea8071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.892+0000 7f8eaccf6640 1 -- 172.21.15.16:0/1611967897 >> v1:172.21.15.16:6789/0 conn(0x7f8ea806e7c0 legacy=0x7f8ea80b4da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.892+0000 7f8eaccf6640 1 -- 172.21.15.16:0/1611967897 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea806da80 con 0x7f8ea8145630 2024-04-24T23:50:52.892 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.892+0000 7f8ea77fe640 1 -- 172.21.15.16:0/1611967897 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1247430854 0 0) 0x7f8ea806d7c0 con 0x7f8ea806e7c0 2024-04-24T23:50:52.893 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.893+0000 7f8eaccf6640 1 --2- 172.21.15.16:0/1611967897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea8145630 0x7f8ea806e280 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0097a0 tx=0x7f8e9c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769befa414b5f350 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.893+0000 7f8ea77fe640 1 -- 172.21.15.16:0/1611967897 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c037070 con 0x7f8ea8145630 2024-04-24T23:50:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.893+0000 7f8ea77fe640 1 -- 172.21.15.16:0/1611967897 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e9c032440 con 0x7f8ea8145630 2024-04-24T23:50:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.893+0000 7f8ea77fe640 1 -- 172.21.15.16:0/1611967897 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c032810 con 0x7f8ea8145630 2024-04-24T23:50:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.893+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1611967897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea8145630 msgr2=0x7f8ea806e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.893+0000 7f8eaef5a640 1 --2- 172.21.15.16:0/1611967897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea8145630 0x7f8ea806e280 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0097a0 tx=0x7f8e9c02a1e0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.893+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1611967897 shutdown_connections 2024-04-24T23:50:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.893+0000 7f8eaef5a640 1 --2- 172.21.15.16:0/1611967897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea8145630 0x7f8ea806e280 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.893+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1611967897 >> 172.21.15.16:0/1611967897 conn(0x7f8ea80cfa20 msgr2=0x7f8ea80cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.893+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1611967897 shutdown_connections 2024-04-24T23:50:52.894 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.893+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1611967897 wait complete. 2024-04-24T23:50:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.895+0000 7f8eaef5a640 1 Processor -- start 2024-04-24T23:50:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.895+0000 7f8eaef5a640 1 -- start start 2024-04-24T23:50:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.895+0000 7f8eaef5a640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea806e7c0 0x7f8ea81404c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.895+0000 7f8eaef5a640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ea80b54b0 0x7f8ea80ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.895+0000 7f8eaef5a640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8ea8145630 0x7f8ea80b0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.895+0000 7f8eaef5a640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea8145b90 con 0x7f8ea806e7c0 2024-04-24T23:50:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.895+0000 7f8eaef5a640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8ea8145a10 con 0x7f8ea80b54b0 2024-04-24T23:50:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.895+0000 7f8eaef5a640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8ea8071940 con 0x7f8ea8145630 2024-04-24T23:50:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8eaccf6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea806e7c0 0x7f8ea81404c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8eaccf6640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea806e7c0 0x7f8ea81404c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49098/0 (socket says 172.21.15.16:49098) 2024-04-24T23:50:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8eaccf6640 1 -- 172.21.15.16:0/1962626479 learned_addr learned my addr 172.21.15.16:0/1962626479 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:52.897 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8ea7fff640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ea80b54b0 0x7f8ea80ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8ead4f7640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8ea8145630 0x7f8ea80b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8eaccf6640 1 -- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8ea8145630 msgr2=0x7f8ea80b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8eaccf6640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8ea8145630 0x7f8ea80b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8eaccf6640 1 -- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ea80b54b0 msgr2=0x7f8ea80ca100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8eaccf6640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ea80b54b0 0x7f8ea80ca100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8eaccf6640 1 -- 172.21.15.16:0/1962626479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ea80c7ed0 con 0x7f8ea806e7c0 2024-04-24T23:50:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8ead4f7640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8ea8145630 0x7f8ea80b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8ea7fff640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8ea80b54b0 0x7f8ea80ca100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8eaccf6640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea806e7c0 0x7f8ea81404c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0097a0 tx=0x7f8e9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8ea57fa640 1 -- 172.21.15.16:0/1962626479 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c037040 con 0x7f8ea806e7c0 2024-04-24T23:50:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8ea57fa640 1 -- 172.21.15.16:0/1962626479 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e9c004250 con 0x7f8ea806e7c0 2024-04-24T23:50:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1962626479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ea80c81b0 con 0x7f8ea806e7c0 2024-04-24T23:50:52.898 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8ea57fa640 1 -- 172.21.15.16:0/1962626479 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c032e20 con 0x7f8ea806e7c0 2024-04-24T23:50:52.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.896+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1962626479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ea80c8770 con 0x7f8ea806e7c0 2024-04-24T23:50:52.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.897+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1962626479 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8ea8068f90 con 0x7f8ea806e7c0 2024-04-24T23:50:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.898+0000 7f8ea57fa640 1 -- 172.21.15.16:0/1962626479 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8e9c030770 con 0x7f8ea806e7c0 2024-04-24T23:50:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.898+0000 7f8ea57fa640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e8c06be60 0x7f8e8c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.898+0000 7f8ea57fa640 1 -- 172.21.15.16:0/1962626479 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f8e9c0abf60 con 0x7f8ea806e7c0 2024-04-24T23:50:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.898+0000 7f8ea57fa640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:6800/1918945722,v1:172.21.15.156:6801/1918945722] conn(0x7f8e8c0723a0 0x7f8e8c0747e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.898+0000 7f8ea57fa640 1 -- 172.21.15.16:0/1962626479 --> [v2:172.21.15.156:6800/1918945722,v1:172.21.15.156:6801/1918945722] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e9c034e90 con 0x7f8e8c0723a0 2024-04-24T23:50:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.898+0000 7f8ea57fa640 1 -- 172.21.15.16:0/1962626479 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f8e9c0ae600 con 0x7f8ea806e7c0 2024-04-24T23:50:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.898+0000 7f8ea7fff640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e8c06be60 0x7f8e8c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.899+0000 7f8ead4f7640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:6800/1918945722,v1:172.21.15.156:6801/1918945722] conn(0x7f8e8c0723a0 0x7f8e8c0747e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:52.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.899+0000 7f8ea7fff640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e8c06be60 0x7f8e8c06e320 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ea815d700 tx=0x7f8e94009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.899+0000 7f8ead4f7640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:6800/1918945722,v1:172.21.15.156:6801/1918945722] conn(0x7f8e8c0723a0 0x7f8e8c0747e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:52.901 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.900+0000 7f8ea57fa640 1 -- 172.21.15.16:0/1962626479 <== osd.5 v2:172.21.15.156:6800/1918945722 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8e9c034e90 con 0x7f8e8c0723a0 2024-04-24T23:50:52.912 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.913+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1962626479 --> [v2:172.21.15.156:6800/1918945722,v1:172.21.15.156:6801/1918945722] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f8ea8070ac0 con 0x7f8e8c0723a0 2024-04-24T23:50:52.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.913+0000 7f8ea57fa640 1 -- 172.21.15.16:0/1962626479 <== osd.5 v2:172.21.15.156:6800/1918945722 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f8ea8070ac0 con 0x7f8e8c0723a0 2024-04-24T23:50:52.913 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:50:52.913 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:50:52.913 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:50:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.914+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:6800/1918945722,v1:172.21.15.156:6801/1918945722] conn(0x7f8e8c0723a0 msgr2=0x7f8e8c0747e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.914+0000 7f8eaef5a640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:6800/1918945722,v1:172.21.15.156:6801/1918945722] conn(0x7f8e8c0723a0 0x7f8e8c0747e0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.914+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e8c06be60 msgr2=0x7f8e8c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.914+0000 7f8eaef5a640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8e8c06be60 0x7f8e8c06e320 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f8ea815d700 tx=0x7f8e94009040 comp rx=0 tx=0).stop 2024-04-24T23:50:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.914+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1962626479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea806e7c0 msgr2=0x7f8ea81404c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:52.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.914+0000 7f8eaef5a640 1 --2- 172.21.15.16:0/1962626479 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ea806e7c0 0x7f8ea81404c0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0097a0 tx=0x7f8e9c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:52.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.914+0000 7f8ead4f7640 1 -- 172.21.15.16:0/1962626479 reap_dead start 2024-04-24T23:50:52.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.914+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1962626479 shutdown_connections 2024-04-24T23:50:52.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.914+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1962626479 >> 172.21.15.16:0/1962626479 conn(0x7f8ea80cfa20 msgr2=0x7f8ea806b800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:52.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.914+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1962626479 shutdown_connections 2024-04-24T23:50:52.915 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:52.914+0000 7f8eaef5a640 1 -- 172.21.15.16:0/1962626479 wait complete. 2024-04-24T23:50:52.924 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-04-24T23:50:53.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.026+0000 7f8f7ff32640 1 Processor -- start 2024-04-24T23:50:53.026 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.026+0000 7f8f7ff32640 1 -- start start 2024-04-24T23:50:53.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.027+0000 7f8f7ff32640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8f7815df40 0x7f8f78072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.027+0000 7f8f7ff32640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f78162a10 con 0x7f8f780727d0 2024-04-24T23:50:53.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.027+0000 7f8f7ff32640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f780ad980 con 0x7f8f78162380 2024-04-24T23:50:53.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.027+0000 7f8f7ff32640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8f780ae2b0 con 0x7f8f7815df40 2024-04-24T23:50:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.027+0000 7f8f7d4cd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8f780727d0 0x7f8f78161c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57670/0 (socket says 172.21.15.16:57670) 2024-04-24T23:50:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.027+0000 7f8f7d4cd640 1 -- 172.21.15.16:0/4138643578 learned_addr learned my addr 172.21.15.16:0/4138643578 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.027+0000 7f8f7dcce640 1 --2- 172.21.15.16:0/4138643578 >> v2:172.21.15.156:3300/0 conn(0x7f8f7815df40 0x7f8f78072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.027+0000 7f8f7cccc640 1 -- 172.21.15.16:0/4138643578 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1137681060 0 0) 0x7f8f78162a10 con 0x7f8f780727d0 2024-04-24T23:50:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.027+0000 7f8f7cccc640 1 -- 172.21.15.16:0/4138643578 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8f64003520 con 0x7f8f780727d0 2024-04-24T23:50:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7cccc640 1 -- 172.21.15.16:0/4138643578 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 305798531 0 0) 0x7f8f64003520 con 0x7f8f780727d0 2024-04-24T23:50:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7cccc640 1 -- 172.21.15.16:0/4138643578 >> v2:172.21.15.156:3300/0 conn(0x7f8f7815df40 msgr2=0x7f8f78072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7cccc640 1 --2- 172.21.15.16:0/4138643578 >> v2:172.21.15.156:3300/0 conn(0x7f8f7815df40 0x7f8f78072290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7cccc640 1 -- 172.21.15.16:0/4138643578 >> v1:172.21.15.156:6789/0 conn(0x7f8f78162380 legacy=0x7f8f780b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7cccc640 1 -- 172.21.15.16:0/4138643578 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f780adc60 con 0x7f8f780727d0 2024-04-24T23:50:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7cccc640 1 -- 172.21.15.16:0/4138643578 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f8f68001670 con 0x7f8f780727d0 2024-04-24T23:50:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7cccc640 1 -- 172.21.15.16:0/4138643578 >> v1:172.21.15.16:6789/0 conn(0x7f8f780727d0 legacy=0x7f8f78161c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.030 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7cccc640 1 --2- 172.21.15.16:0/4138643578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f640039b0 0x7f8f64003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7cccc640 1 --2- 172.21.15.16:0/4138643578 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f64004800 0x7f8f64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7cccc640 1 --2- 172.21.15.16:0/4138643578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f640051c0 0x7f8f6400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7cccc640 1 -- 172.21.15.16:0/4138643578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8f780ae2b0 con 0x7f8f640039b0 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7cccc640 1 -- 172.21.15.16:0/4138643578 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8f6400c060 con 0x7f8f640051c0 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7cccc640 1 -- 172.21.15.16:0/4138643578 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8f6400c260 con 0x7f8f64004800 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7dcce640 1 --2- 172.21.15.16:0/4138643578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f640039b0 0x7f8f64003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7d4cd640 1 --2- 172.21.15.16:0/4138643578 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f64004800 0x7f8f64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7e4cf640 1 --2- 172.21.15.16:0/4138643578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f640051c0 0x7f8f6400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7dcce640 1 -- 172.21.15.16:0/4138643578 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f64004800 msgr2=0x7f8f64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7dcce640 1 --2- 172.21.15.16:0/4138643578 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f64004800 0x7f8f64004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7dcce640 1 -- 172.21.15.16:0/4138643578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f640051c0 msgr2=0x7f8f6400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7dcce640 1 --2- 172.21.15.16:0/4138643578 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f640051c0 0x7f8f6400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7dcce640 1 -- 172.21.15.16:0/4138643578 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f64003520 con 0x7f8f640039b0 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.028+0000 7f8f7e4cf640 1 -- 172.21.15.16:0/4138643578 reap_dead start 2024-04-24T23:50:53.031 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.029+0000 7f8f7dcce640 1 --2- 172.21.15.16:0/4138643578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f640039b0 0x7f8f64003dd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8f74009ac0 tx=0x7f8f64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3bb69ba3b706d8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.029+0000 7f8f7cccc640 1 -- 172.21.15.16:0/4138643578 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f74036070 con 0x7f8f640039b0 2024-04-24T23:50:53.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.029+0000 7f8f7cccc640 1 -- 172.21.15.16:0/4138643578 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f74031400 con 0x7f8f640039b0 2024-04-24T23:50:53.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.029+0000 7f8f7cccc640 1 -- 172.21.15.16:0/4138643578 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f74009690 con 0x7f8f640039b0 2024-04-24T23:50:53.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.029+0000 7f8f7ff32640 1 -- 172.21.15.16:0/4138643578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f640039b0 msgr2=0x7f8f64003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.029+0000 7f8f7ff32640 1 --2- 172.21.15.16:0/4138643578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f640039b0 0x7f8f64003dd0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8f74009ac0 tx=0x7f8f64005830 comp rx=0 tx=0).stop 2024-04-24T23:50:53.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.029+0000 7f8f7ff32640 1 -- 172.21.15.16:0/4138643578 shutdown_connections 2024-04-24T23:50:53.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.029+0000 7f8f7ff32640 1 --2- 172.21.15.16:0/4138643578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f640039b0 0x7f8f64003dd0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.029+0000 7f8f7ff32640 1 -- 172.21.15.16:0/4138643578 >> 172.21.15.16:0/4138643578 conn(0x7f8f780d0260 msgr2=0x7f8f780d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:53.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.030+0000 7f8f7ff32640 1 -- 172.21.15.16:0/4138643578 shutdown_connections 2024-04-24T23:50:53.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.030+0000 7f8f7ff32640 1 -- 172.21.15.16:0/4138643578 wait complete. 2024-04-24T23:50:53.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.031+0000 7f8f7ff32640 1 Processor -- start 2024-04-24T23:50:53.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.031+0000 7f8f7ff32640 1 -- start start 2024-04-24T23:50:53.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.031+0000 7f8f7ff32640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f640039b0 0x7f8f7815ff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.031+0000 7f8f7ff32640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f781f7e20 0x7f8f78156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.031+0000 7f8f7ff32640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f781f82a0 0x7f8f7806c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.031+0000 7f8f7ff32640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8f780ae740 con 0x7f8f640039b0 2024-04-24T23:50:53.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.031+0000 7f8f7ff32640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8f781627b0 con 0x7f8f781f82a0 2024-04-24T23:50:53.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.031+0000 7f8f7ff32640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8f7815e320 con 0x7f8f781f7e20 2024-04-24T23:50:53.035 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f7dcce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f640039b0 0x7f8f7815ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f7dcce640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f640039b0 0x7f8f7815ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49104/0 (socket says 172.21.15.16:49104) 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f7dcce640 1 -- 172.21.15.16:0/344139735 learned_addr learned my addr 172.21.15.16:0/344139735 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f7d4cd640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f781f7e20 0x7f8f78156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f7e4cf640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f781f82a0 0x7f8f7806c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f7dcce640 1 -- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f781f7e20 msgr2=0x7f8f78156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f7dcce640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f781f7e20 0x7f8f78156790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f7dcce640 1 -- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f781f82a0 msgr2=0x7f8f7806c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f7dcce640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f781f82a0 0x7f8f7806c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f7dcce640 1 -- 172.21.15.16:0/344139735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f781f87e0 con 0x7f8f640039b0 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f7d4cd640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f781f7e20 0x7f8f78156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f7dcce640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f640039b0 0x7f8f7815ff10 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8f74034040 tx=0x7f8f74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f6e7fc640 1 -- 172.21.15.16:0/344139735 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7403f020 con 0x7f8f640039b0 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f6e7fc640 1 -- 172.21.15.16:0/344139735 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f74037040 con 0x7f8f640039b0 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f6e7fc640 1 -- 172.21.15.16:0/344139735 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f74036070 con 0x7f8f640039b0 2024-04-24T23:50:53.036 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f7ff32640 1 -- 172.21.15.16:0/344139735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f781f8ac0 con 0x7f8f640039b0 2024-04-24T23:50:53.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.032+0000 7f8f7ff32640 1 -- 172.21.15.16:0/344139735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f7806be00 con 0x7f8f640039b0 2024-04-24T23:50:53.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.033+0000 7f8f7ff32640 1 -- 172.21.15.16:0/344139735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f7806e5c0 con 0x7f8f640039b0 2024-04-24T23:50:53.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.034+0000 7f8f6e7fc640 1 -- 172.21.15.16:0/344139735 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8f74042020 con 0x7f8f640039b0 2024-04-24T23:50:53.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.034+0000 7f8f6e7fc640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8f5c06be60 0x7f8f5c06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.034+0000 7f8f6e7fc640 1 -- 172.21.15.16:0/344139735 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f8f740aaca0 con 0x7f8f640039b0 2024-04-24T23:50:53.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.034+0000 7f8f7d4cd640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8f5c06be60 0x7f8f5c06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.035+0000 7f8f7d4cd640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8f5c06be60 0x7f8f5c06e320 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8f68001eb0 tx=0x7f8f68009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.038+0000 7f8f6e7fc640 1 -- 172.21.15.16:0/344139735 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8f74049030 con 0x7f8f640039b0 2024-04-24T23:50:53.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.222+0000 7f8f7ff32640 1 -- 172.21.15.16:0/344139735 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f8f78159d80 con 0x7f8f5c06be60 2024-04-24T23:50:53.222 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.223+0000 7f8f6e7fc640 1 -- 172.21.15.16:0/344139735 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8f78159d80 con 0x7f8f5c06be60 2024-04-24T23:50:53.223 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 5 to deep-scrub 2024-04-24T23:50:53.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.225+0000 7f8f7ff32640 1 -- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8f5c06be60 msgr2=0x7f8f5c06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.225+0000 7f8f7ff32640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8f5c06be60 0x7f8f5c06e320 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8f68001eb0 tx=0x7f8f68009040 comp rx=0 tx=0).stop 2024-04-24T23:50:53.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.225+0000 7f8f7ff32640 1 -- 172.21.15.16:0/344139735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f640039b0 msgr2=0x7f8f7815ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.225+0000 7f8f7ff32640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f640039b0 0x7f8f7815ff10 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8f74034040 tx=0x7f8f74004730 comp rx=0 tx=0).stop 2024-04-24T23:50:53.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.225+0000 7f8f7ff32640 1 -- 172.21.15.16:0/344139735 shutdown_connections 2024-04-24T23:50:53.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.225+0000 7f8f7ff32640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8f5c06be60 0x7f8f5c06e320 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.225+0000 7f8f7ff32640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f781f82a0 0x7f8f7806c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.225 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.225+0000 7f8f7ff32640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8f781f7e20 0x7f8f78156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.225+0000 7f8f7ff32640 1 --2- 172.21.15.16:0/344139735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8f640039b0 0x7f8f7815ff10 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.225+0000 7f8f7ff32640 1 -- 172.21.15.16:0/344139735 >> 172.21.15.16:0/344139735 conn(0x7f8f780d0260 msgr2=0x7f8f780b0180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:53.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.225+0000 7f8f7ff32640 1 -- 172.21.15.16:0/344139735 shutdown_connections 2024-04-24T23:50:53.226 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.225+0000 7f8f7ff32640 1 -- 172.21.15.16:0/344139735 wait complete. 2024-04-24T23:50:53.235 INFO:tasks.ceph:Scrubbing osd.6 2024-04-24T23:50:53.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:50:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.340+0000 7f2ebe770640 1 Processor -- start 2024-04-24T23:50:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.340+0000 7f2ebe770640 1 -- start start 2024-04-24T23:50:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.341+0000 7f2ebe770640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f2eb8071940 0x7f2eb80b5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.341+0000 7f2ebe770640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eb806d7c0 con 0x7f2eb80b4da0 2024-04-24T23:50:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.341+0000 7f2ebe770640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eb806e990 con 0x7f2eb8145630 2024-04-24T23:50:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.341+0000 7f2ebe770640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f2eb80b5750 con 0x7f2eb8071940 2024-04-24T23:50:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.341+0000 7f2eb77fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2eb80b4da0 0x7f2eb8071230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57672/0 (socket says 172.21.15.16:57672) 2024-04-24T23:50:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.341+0000 7f2eb77fe640 1 -- 172.21.15.16:0/2078825244 learned_addr learned my addr 172.21.15.16:0/2078825244 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.341+0000 7f2eb6ffd640 1 -- 172.21.15.16:0/2078825244 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3664966533 0 0) 0x7f2eb806d7c0 con 0x7f2eb80b4da0 2024-04-24T23:50:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.341+0000 7f2ebcd0d640 1 --2- 172.21.15.16:0/2078825244 >> v2:172.21.15.156:3300/0 conn(0x7f2eb8071940 0x7f2eb80b5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.341+0000 7f2eb6ffd640 1 -- 172.21.15.16:0/2078825244 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2ea4003520 con 0x7f2eb80b4da0 2024-04-24T23:50:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb6ffd640 1 -- 172.21.15.16:0/2078825244 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2066078035 0 0) 0x7f2ea4003520 con 0x7f2eb80b4da0 2024-04-24T23:50:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb6ffd640 1 -- 172.21.15.16:0/2078825244 >> v2:172.21.15.156:3300/0 conn(0x7f2eb8071940 msgr2=0x7f2eb80b5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb6ffd640 1 --2- 172.21.15.16:0/2078825244 >> v2:172.21.15.156:3300/0 conn(0x7f2eb8071940 0x7f2eb80b5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb6ffd640 1 -- 172.21.15.16:0/2078825244 >> v1:172.21.15.156:6789/0 conn(0x7f2eb8145630 legacy=0x7f2eb806e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb6ffd640 1 -- 172.21.15.16:0/2078825244 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb8160660 con 0x7f2eb80b4da0 2024-04-24T23:50:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb6ffd640 1 -- 172.21.15.16:0/2078825244 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f2ea8001670 con 0x7f2eb80b4da0 2024-04-24T23:50:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb6ffd640 1 -- 172.21.15.16:0/2078825244 >> v1:172.21.15.16:6789/0 conn(0x7f2eb80b4da0 legacy=0x7f2eb8071230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb6ffd640 1 --2- 172.21.15.16:0/2078825244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ea40039b0 0x7f2ea4003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb6ffd640 1 --2- 172.21.15.16:0/2078825244 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ea4004800 0x7f2ea4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb6ffd640 1 --2- 172.21.15.16:0/2078825244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ea40051c0 0x7f2ea400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb6ffd640 1 -- 172.21.15.16:0/2078825244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb80b5750 con 0x7f2ea40051c0 2024-04-24T23:50:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb6ffd640 1 -- 172.21.15.16:0/2078825244 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2ea400c040 con 0x7f2ea40039b0 2024-04-24T23:50:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb6ffd640 1 -- 172.21.15.16:0/2078825244 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2ea400c240 con 0x7f2ea4004800 2024-04-24T23:50:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2ebcd0d640 1 --2- 172.21.15.16:0/2078825244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ea40051c0 0x7f2ea400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb77fe640 1 --2- 172.21.15.16:0/2078825244 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ea4004800 0x7f2ea4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb7fff640 1 --2- 172.21.15.16:0/2078825244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ea40039b0 0x7f2ea4003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2ebcd0d640 1 -- 172.21.15.16:0/2078825244 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ea4004800 msgr2=0x7f2ea4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2ebcd0d640 1 --2- 172.21.15.16:0/2078825244 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2ea4004800 0x7f2ea4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2ebcd0d640 1 -- 172.21.15.16:0/2078825244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ea40039b0 msgr2=0x7f2ea4003db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2ebcd0d640 1 --2- 172.21.15.16:0/2078825244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ea40039b0 0x7f2ea4003db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2ebcd0d640 1 -- 172.21.15.16:0/2078825244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea4003520 con 0x7f2ea40051c0 2024-04-24T23:50:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.342+0000 7f2eb7fff640 1 --2- 172.21.15.16:0/2078825244 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ea40039b0 0x7f2ea4003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.343+0000 7f2ebcd0d640 1 -- 172.21.15.16:0/2078825244 reap_dead start 2024-04-24T23:50:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.343+0000 7f2ebcd0d640 1 --2- 172.21.15.16:0/2078825244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ea40051c0 0x7f2ea400b9c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009fc0 tx=0x7f2ea4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=804eee997aeb9261 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.343+0000 7f2eb6ffd640 1 -- 172.21.15.16:0/2078825244 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0035020 con 0x7f2ea40051c0 2024-04-24T23:50:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.343+0000 7f2eb6ffd640 1 -- 172.21.15.16:0/2078825244 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eb0003bd0 con 0x7f2ea40051c0 2024-04-24T23:50:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.343+0000 7f2eb6ffd640 1 -- 172.21.15.16:0/2078825244 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0009c10 con 0x7f2ea40051c0 2024-04-24T23:50:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.343+0000 7f2ebe770640 1 -- 172.21.15.16:0/2078825244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ea40051c0 msgr2=0x7f2ea400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.343+0000 7f2ebe770640 1 --2- 172.21.15.16:0/2078825244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ea40051c0 0x7f2ea400b9c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0009fc0 tx=0x7f2ea4005830 comp rx=0 tx=0).stop 2024-04-24T23:50:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.343+0000 7f2ebe770640 1 -- 172.21.15.16:0/2078825244 shutdown_connections 2024-04-24T23:50:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.343+0000 7f2ebe770640 1 --2- 172.21.15.16:0/2078825244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ea40051c0 0x7f2ea400b9c0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.343+0000 7f2ebe770640 1 -- 172.21.15.16:0/2078825244 >> 172.21.15.16:0/2078825244 conn(0x7f2eb80cfa20 msgr2=0x7f2eb80cfe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.344+0000 7f2ebe770640 1 -- 172.21.15.16:0/2078825244 shutdown_connections 2024-04-24T23:50:53.348 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.344+0000 7f2ebe770640 1 -- 172.21.15.16:0/2078825244 wait complete. 2024-04-24T23:50:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.345+0000 7f2ebe770640 1 Processor -- start 2024-04-24T23:50:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.345+0000 7f2ebe770640 1 -- start start 2024-04-24T23:50:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebe770640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ea40051c0 0x7f2eb813bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebe770640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eb80cbe20 0x7f2eb815ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebe770640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb813d7e0 0x7f2eb813dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebe770640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb806dd60 con 0x7f2eb813d7e0 2024-04-24T23:50:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebe770640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb806d560 con 0x7f2ea40051c0 2024-04-24T23:50:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebe770640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f2eb8145a10 con 0x7f2eb80cbe20 2024-04-24T23:50:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebcd0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb813d7e0 0x7f2eb813dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebcd0d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb813d7e0 0x7f2eb813dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49118/0 (socket says 172.21.15.16:49118) 2024-04-24T23:50:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebcd0d640 1 -- 172.21.15.16:0/1070945332 learned_addr learned my addr 172.21.15.16:0/1070945332 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2eb7fff640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ea40051c0 0x7f2eb813bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2eb77fe640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eb80cbe20 0x7f2eb815ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebcd0d640 1 -- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eb80cbe20 msgr2=0x7f2eb815ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebcd0d640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eb80cbe20 0x7f2eb815ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebcd0d640 1 -- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ea40051c0 msgr2=0x7f2eb813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebcd0d640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ea40051c0 0x7f2eb813bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebcd0d640 1 -- 172.21.15.16:0/1070945332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb813c140 con 0x7f2eb813d7e0 2024-04-24T23:50:53.353 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2eb7fff640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2ea40051c0 0x7f2eb813bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2eb77fe640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f2eb80cbe20 0x7f2eb815ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:50:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebcd0d640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb813d7e0 0x7f2eb813dc90 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0032d80 tx=0x7f2eb0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2eb4ff9640 1 -- 172.21.15.16:0/1070945332 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb0035030 con 0x7f2eb813d7e0 2024-04-24T23:50:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2eb4ff9640 1 -- 172.21.15.16:0/1070945332 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eb0043070 con 0x7f2eb813d7e0 2024-04-24T23:50:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebe770640 1 -- 172.21.15.16:0/1070945332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb813c420 con 0x7f2eb813d7e0 2024-04-24T23:50:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2eb4ff9640 1 -- 172.21.15.16:0/1070945332 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eb002d070 con 0x7f2eb813d7e0 2024-04-24T23:50:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.346+0000 7f2ebe770640 1 -- 172.21.15.16:0/1070945332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb813c9e0 con 0x7f2eb813d7e0 2024-04-24T23:50:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.347+0000 7f2ebe770640 1 -- 172.21.15.16:0/1070945332 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2eb81f1810 con 0x7f2eb813d7e0 2024-04-24T23:50:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.348+0000 7f2eb4ff9640 1 -- 172.21.15.16:0/1070945332 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2eb0005d10 con 0x7f2eb813d7e0 2024-04-24T23:50:53.354 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.348+0000 7f2eb4ff9640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e9806be60 0x7f2e9806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.348+0000 7f2eb4ff9640 1 -- 172.21.15.16:0/1070945332 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f2eb00ab3f0 con 0x7f2eb813d7e0 2024-04-24T23:50:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.348+0000 7f2eb4ff9640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:6816/667041555,v1:172.21.15.156:6817/667041555] conn(0x7f2e980723a0 0x7f2e980747e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.348+0000 7f2eb4ff9640 1 -- 172.21.15.16:0/1070945332 --> [v2:172.21.15.156:6816/667041555,v1:172.21.15.156:6817/667041555] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2eb0033e50 con 0x7f2e980723a0 2024-04-24T23:50:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.348+0000 7f2eb4ff9640 1 -- 172.21.15.16:0/1070945332 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f2eb00b2050 con 0x7f2eb813d7e0 2024-04-24T23:50:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.349+0000 7f2eb7fff640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e9806be60 0x7f2e9806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.349+0000 7f2eb77fe640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:6816/667041555,v1:172.21.15.156:6817/667041555] conn(0x7f2e980723a0 0x7f2e980747e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.349+0000 7f2eb7fff640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e9806be60 0x7f2e9806e320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2eb806b500 tx=0x7f2ea0009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.349+0000 7f2eb77fe640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:6816/667041555,v1:172.21.15.156:6817/667041555] conn(0x7f2e980723a0 0x7f2e980747e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.355 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.350+0000 7f2eb4ff9640 1 -- 172.21.15.16:0/1070945332 <== osd.6 v2:172.21.15.156:6816/667041555 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2eb0033e50 con 0x7f2e980723a0 2024-04-24T23:50:53.362 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.363+0000 7f2ebe770640 1 -- 172.21.15.16:0/1070945332 --> [v2:172.21.15.156:6816/667041555,v1:172.21.15.156:6817/667041555] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2eb8071760 con 0x7f2e980723a0 2024-04-24T23:50:53.363 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.364+0000 7f2eb4ff9640 1 -- 172.21.15.16:0/1070945332 <== osd.6 v2:172.21.15.156:6816/667041555 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2eb8071760 con 0x7f2e980723a0 2024-04-24T23:50:53.363 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:50:53.363 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:50:53.363 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:50:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.364+0000 7f2ebe770640 1 -- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:6816/667041555,v1:172.21.15.156:6817/667041555] conn(0x7f2e980723a0 msgr2=0x7f2e980747e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.364+0000 7f2ebe770640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:6816/667041555,v1:172.21.15.156:6817/667041555] conn(0x7f2e980723a0 0x7f2e980747e0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.364+0000 7f2ebe770640 1 -- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e9806be60 msgr2=0x7f2e9806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.364+0000 7f2ebe770640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f2e9806be60 0x7f2e9806e320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2eb806b500 tx=0x7f2ea0009040 comp rx=0 tx=0).stop 2024-04-24T23:50:53.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.364+0000 7f2ebe770640 1 -- 172.21.15.16:0/1070945332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb813d7e0 msgr2=0x7f2eb813dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.364+0000 7f2ebe770640 1 --2- 172.21.15.16:0/1070945332 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2eb813d7e0 0x7f2eb813dc90 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2eb0032d80 tx=0x7f2eb0009750 comp rx=0 tx=0).stop 2024-04-24T23:50:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.364+0000 7f2ebcd0d640 1 -- 172.21.15.16:0/1070945332 reap_dead start 2024-04-24T23:50:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.364+0000 7f2ebe770640 1 -- 172.21.15.16:0/1070945332 shutdown_connections 2024-04-24T23:50:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.364+0000 7f2ebe770640 1 -- 172.21.15.16:0/1070945332 >> 172.21.15.16:0/1070945332 conn(0x7f2eb80cfa20 msgr2=0x7f2eb806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.364+0000 7f2ebe770640 1 -- 172.21.15.16:0/1070945332 shutdown_connections 2024-04-24T23:50:53.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.364+0000 7f2ebe770640 1 -- 172.21.15.16:0/1070945332 wait complete. 2024-04-24T23:50:53.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-04-24T23:50:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.476+0000 7f8fd73b0640 1 Processor -- start 2024-04-24T23:50:53.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.476+0000 7f8fd73b0640 1 -- start start 2024-04-24T23:50:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fd73b0640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f8fd015dfa0 0x7f8fd00d0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fd73b0640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fd00ce5f0 0x7f8fd00cea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fd73b0640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fd00d05a0 con 0x7f8fd00b5f00 2024-04-24T23:50:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fd73b0640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f8fd00b7360 con 0x7f8fd00ce5f0 2024-04-24T23:50:53.484 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fd73b0640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f8fd00cc640 con 0x7f8fd015dfa0 2024-04-24T23:50:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fd494b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8fd00ce5f0 0x7f8fd00cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fd594d640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8fd00b5f00 0x7f8fd00b6370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57674/0 (socket says 172.21.15.16:57674) 2024-04-24T23:50:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fd594d640 1 -- 172.21.15.16:0/3080119561 learned_addr learned my addr 172.21.15.16:0/3080119561 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fd514c640 1 --2- 172.21.15.16:0/3080119561 >> v2:172.21.15.156:3300/0 conn(0x7f8fd015dfa0 0x7f8fd00d0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fd494b640 1 -- 172.21.15.16:0/3080119561 >> v2:172.21.15.156:3300/0 conn(0x7f8fd015dfa0 msgr2=0x7f8fd00d0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fd494b640 1 --2- 172.21.15.16:0/3080119561 >> v2:172.21.15.156:3300/0 conn(0x7f8fd015dfa0 0x7f8fd00d0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fd494b640 1 -- 172.21.15.16:0/3080119561 >> v1:172.21.15.16:6789/0 conn(0x7f8fd00b5f00 legacy=0x7f8fd00b6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fc7fff640 1 -- 172.21.15.16:0/3080119561 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 122379039 0 0) 0x7f8fd00d05a0 con 0x7f8fd00b5f00 2024-04-24T23:50:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fd494b640 1 -- 172.21.15.16:0/3080119561 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd00d0880 con 0x7f8fd00ce5f0 2024-04-24T23:50:53.485 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.477+0000 7f8fd514c640 1 --2- 172.21.15.16:0/3080119561 >> v2:172.21.15.156:3300/0 conn(0x7f8fd015dfa0 0x7f8fd00d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:53.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.478+0000 7f8fd494b640 1 --2- 172.21.15.16:0/3080119561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd00ce5f0 0x7f8fd00cea70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00cf3e0 tx=0x7f8fc002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0c6b4e9a8067a1a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.478+0000 7f8fc7fff640 1 -- 172.21.15.16:0/3080119561 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0037070 con 0x7f8fd00ce5f0 2024-04-24T23:50:53.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.478+0000 7f8fc7fff640 1 -- 172.21.15.16:0/3080119561 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fc0032440 con 0x7f8fd00ce5f0 2024-04-24T23:50:53.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.478+0000 7f8fc7fff640 1 -- 172.21.15.16:0/3080119561 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc0032810 con 0x7f8fd00ce5f0 2024-04-24T23:50:53.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.479+0000 7f8fd73b0640 1 -- 172.21.15.16:0/3080119561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd00ce5f0 msgr2=0x7f8fd00cea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.479+0000 7f8fd73b0640 1 --2- 172.21.15.16:0/3080119561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd00ce5f0 0x7f8fd00cea70 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8fd00cf3e0 tx=0x7f8fc002a1a0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.479+0000 7f8fd73b0640 1 -- 172.21.15.16:0/3080119561 shutdown_connections 2024-04-24T23:50:53.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.479+0000 7f8fd73b0640 1 --2- 172.21.15.16:0/3080119561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd00ce5f0 0x7f8fd00cea70 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.479+0000 7f8fd73b0640 1 --2- 172.21.15.16:0/3080119561 >> v2:172.21.15.156:3300/0 conn(0x7f8fd015dfa0 0x7f8fd00d0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.479+0000 7f8fd73b0640 1 -- 172.21.15.16:0/3080119561 >> 172.21.15.16:0/3080119561 conn(0x7f8fd006b4d0 msgr2=0x7f8fd00b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:53.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.479+0000 7f8fd73b0640 1 -- 172.21.15.16:0/3080119561 shutdown_connections 2024-04-24T23:50:53.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.479+0000 7f8fd73b0640 1 -- 172.21.15.16:0/3080119561 wait complete. 2024-04-24T23:50:53.486 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.480+0000 7f8fd73b0640 1 Processor -- start 2024-04-24T23:50:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.480+0000 7f8fd73b0640 1 -- start start 2024-04-24T23:50:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd73b0640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fd00b5f00 0x7f8fd0153420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd73b0640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fd00ce5f0 0x7f8fd00ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd73b0640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd015dfa0 0x7f8fd015a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd73b0640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd015e380 con 0x7f8fd015dfa0 2024-04-24T23:50:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd73b0640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd00cc2d0 con 0x7f8fd00b5f00 2024-04-24T23:50:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd73b0640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f8fd00d1520 con 0x7f8fd00ce5f0 2024-04-24T23:50:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd594d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd015dfa0 0x7f8fd015a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd594d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd015dfa0 0x7f8fd015a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49136/0 (socket says 172.21.15.16:49136) 2024-04-24T23:50:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd594d640 1 -- 172.21.15.16:0/1573195157 learned_addr learned my addr 172.21.15.16:0/1573195157 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd494b640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fd00ce5f0 0x7f8fd00ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd514c640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fd00b5f00 0x7f8fd0153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd594d640 1 -- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fd00ce5f0 msgr2=0x7f8fd00ca0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd594d640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fd00ce5f0 0x7f8fd00ca0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd594d640 1 -- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fd00b5f00 msgr2=0x7f8fd0153420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd594d640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fd00b5f00 0x7f8fd0153420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd594d640 1 -- 172.21.15.16:0/1573195157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd00c9050 con 0x7f8fd015dfa0 2024-04-24T23:50:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd514c640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fd00b5f00 0x7f8fd0153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd594d640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd015dfa0 0x7f8fd015a3c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00c7a0 tx=0x7f8fcc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fc5ffb640 1 -- 172.21.15.16:0/1573195157 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc018070 con 0x7f8fd015dfa0 2024-04-24T23:50:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fc5ffb640 1 -- 172.21.15.16:0/1573195157 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fcc00a5e0 con 0x7f8fd015dfa0 2024-04-24T23:50:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fd73b0640 1 -- 172.21.15.16:0/1573195157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd00c9330 con 0x7f8fd015dfa0 2024-04-24T23:50:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.481+0000 7f8fc5ffb640 1 -- 172.21.15.16:0/1573195157 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc00aa50 con 0x7f8fd015dfa0 2024-04-24T23:50:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.482+0000 7f8fd73b0640 1 -- 172.21.15.16:0/1573195157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd00c98f0 con 0x7f8fd015dfa0 2024-04-24T23:50:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.483+0000 7f8fd73b0640 1 -- 172.21.15.16:0/1573195157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fd006bea0 con 0x7f8fd015dfa0 2024-04-24T23:50:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.483+0000 7f8fc5ffb640 1 -- 172.21.15.16:0/1573195157 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8fcc01b070 con 0x7f8fd015dfa0 2024-04-24T23:50:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.483+0000 7f8fc5ffb640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8fb406be30 0x7f8fb406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.483+0000 7f8fc5ffb640 1 -- 172.21.15.16:0/1573195157 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f8fcc08a370 con 0x7f8fd015dfa0 2024-04-24T23:50:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.484+0000 7f8fd514c640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8fb406be30 0x7f8fb406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.484+0000 7f8fd514c640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8fb406be30 0x7f8fb406e2f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8006fd0 tx=0x7f8fc8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.488+0000 7f8fc5ffb640 1 -- 172.21.15.16:0/1573195157 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f8fcc058e40 con 0x7f8fd015dfa0 2024-04-24T23:50:53.668 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.669+0000 7f8fd73b0640 1 -- 172.21.15.16:0/1573195157 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7f8fd004f5e0 con 0x7f8fb406be30 2024-04-24T23:50:53.669 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.669+0000 7f8fc5ffb640 1 -- 172.21.15.16:0/1573195157 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8fd004f5e0 con 0x7f8fb406be30 2024-04-24T23:50:53.669 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 6 to deep-scrub 2024-04-24T23:50:53.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.671+0000 7f8fd73b0640 1 -- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8fb406be30 msgr2=0x7f8fb406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.671 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.671+0000 7f8fd73b0640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8fb406be30 0x7f8fb406e2f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8006fd0 tx=0x7f8fc8008040 comp rx=0 tx=0).stop 2024-04-24T23:50:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.671+0000 7f8fd73b0640 1 -- 172.21.15.16:0/1573195157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd015dfa0 msgr2=0x7f8fd015a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.671+0000 7f8fd73b0640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd015dfa0 0x7f8fd015a3c0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc00c7a0 tx=0x7f8fcc00cc60 comp rx=0 tx=0).stop 2024-04-24T23:50:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.672+0000 7f8fd73b0640 1 -- 172.21.15.16:0/1573195157 shutdown_connections 2024-04-24T23:50:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.672+0000 7f8fd73b0640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f8fb406be30 0x7f8fb406e2f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.672+0000 7f8fd73b0640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8fd015dfa0 0x7f8fd015a3c0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.672+0000 7f8fd73b0640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f8fd00ce5f0 0x7f8fd00ca0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.672+0000 7f8fd73b0640 1 --2- 172.21.15.16:0/1573195157 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fd00b5f00 0x7f8fd0153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.672+0000 7f8fd73b0640 1 -- 172.21.15.16:0/1573195157 >> 172.21.15.16:0/1573195157 conn(0x7f8fd006b4d0 msgr2=0x7f8fd015fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.672+0000 7f8fd73b0640 1 -- 172.21.15.16:0/1573195157 shutdown_connections 2024-04-24T23:50:53.672 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.672+0000 7f8fd73b0640 1 -- 172.21.15.16:0/1573195157 wait complete. 2024-04-24T23:50:53.682 INFO:tasks.ceph:Scrubbing osd.7 2024-04-24T23:50:53.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-04-24T23:50:53.786 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.787+0000 7f33b4038640 1 Processor -- start 2024-04-24T23:50:53.787 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.787+0000 7f33b4038640 1 -- start start 2024-04-24T23:50:53.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.788+0000 7f33b4038640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f33ac15e2b0 0x7f33ac0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.788+0000 7f33b4038640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33ac0b4a50 0x7f33ac0b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.788+0000 7f33b4038640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33ac0cc670 con 0x7f33ac1627a0 2024-04-24T23:50:53.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.788+0000 7f33b4038640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f33ac15e690 con 0x7f33ac15e2b0 2024-04-24T23:50:53.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.788+0000 7f33b4038640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f33ac0b5400 con 0x7f33ac0b4a50 2024-04-24T23:50:53.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.788+0000 7f33b1dd4640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f33ac15e2b0 0x7f33ac0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.788+0000 7f33b15d3640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f33ac1627a0 0x7f33ac0cd3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57688/0 (socket says 172.21.15.16:57688) 2024-04-24T23:50:53.788 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.788+0000 7f33b15d3640 1 -- 172.21.15.16:0/909208143 learned_addr learned my addr 172.21.15.16:0/909208143 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:53.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.788+0000 7f33b25d5640 1 --2- 172.21.15.16:0/909208143 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33ac0b4a50 0x7f33ac0b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.788+0000 7f33b1dd4640 1 -- 172.21.15.16:0/909208143 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33ac0b4a50 msgr2=0x7f33ac0b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:53.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.788+0000 7f33b1dd4640 1 --2- 172.21.15.16:0/909208143 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33ac0b4a50 0x7f33ac0b4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.788+0000 7f33b1dd4640 1 -- 172.21.15.16:0/909208143 >> v1:172.21.15.16:6789/0 conn(0x7f33ac1627a0 legacy=0x7f33ac0cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.789 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.788+0000 7f33b1dd4640 1 -- 172.21.15.16:0/909208143 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33ac0cdad0 con 0x7f33ac15e2b0 2024-04-24T23:50:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.789+0000 7f33b1dd4640 1 --2- 172.21.15.16:0/909208143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac15e2b0 0x7f33ac0711b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f33a80097d0 tx=0x7f33a802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2957c446dccfbf3b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.789+0000 7f33b0dd2640 1 -- 172.21.15.16:0/909208143 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a8037070 con 0x7f33ac15e2b0 2024-04-24T23:50:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.789+0000 7f33b0dd2640 1 -- 172.21.15.16:0/909208143 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a8032440 con 0x7f33ac15e2b0 2024-04-24T23:50:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.789+0000 7f33b0dd2640 1 -- 172.21.15.16:0/909208143 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a8032810 con 0x7f33ac15e2b0 2024-04-24T23:50:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.790+0000 7f33b4038640 1 -- 172.21.15.16:0/909208143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac15e2b0 msgr2=0x7f33ac0711b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.790+0000 7f33b4038640 1 --2- 172.21.15.16:0/909208143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac15e2b0 0x7f33ac0711b0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f33a80097d0 tx=0x7f33a802a1c0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.790+0000 7f33b4038640 1 -- 172.21.15.16:0/909208143 shutdown_connections 2024-04-24T23:50:53.790 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.790+0000 7f33b4038640 1 --2- 172.21.15.16:0/909208143 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33ac0b4a50 0x7f33ac0b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.790+0000 7f33b4038640 1 --2- 172.21.15.16:0/909208143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac15e2b0 0x7f33ac0711b0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.790+0000 7f33b4038640 1 -- 172.21.15.16:0/909208143 >> 172.21.15.16:0/909208143 conn(0x7f33ac0d1bc0 msgr2=0x7f33ac0d1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:53.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.790+0000 7f33b4038640 1 -- 172.21.15.16:0/909208143 shutdown_connections 2024-04-24T23:50:53.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.790+0000 7f33b4038640 1 -- 172.21.15.16:0/909208143 wait complete. 2024-04-24T23:50:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.791+0000 7f33b4038640 1 Processor -- start 2024-04-24T23:50:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.791+0000 7f33b4038640 1 -- start start 2024-04-24T23:50:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b4038640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33ac0b4a50 0x7f33ac0aed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b4038640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33ac15e2b0 0x7f33ac153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b4038640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac1f4410 0x7f33ac0c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b4038640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac0ccbc0 con 0x7f33ac1f4410 2024-04-24T23:50:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b4038640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f33ac0ccf30 con 0x7f33ac15e2b0 2024-04-24T23:50:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b4038640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f33ac162c30 con 0x7f33ac0b4a50 2024-04-24T23:50:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b25d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac1f4410 0x7f33ac0c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b25d5640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac1f4410 0x7f33ac0c7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49158/0 (socket says 172.21.15.16:49158) 2024-04-24T23:50:53.793 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b25d5640 1 -- 172.21.15.16:0/1217723907 learned_addr learned my addr 172.21.15.16:0/1217723907 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b1dd4640 1 --2- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33ac0b4a50 0x7f33ac0aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b15d3640 1 --2- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33ac15e2b0 0x7f33ac153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b25d5640 1 -- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33ac0b4a50 msgr2=0x7f33ac0aed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b25d5640 1 --2- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f33ac0b4a50 0x7f33ac0aed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b25d5640 1 -- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33ac15e2b0 msgr2=0x7f33ac153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b25d5640 1 --2- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f33ac15e2b0 0x7f33ac153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b25d5640 1 -- 172.21.15.16:0/1217723907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33ac0c8a80 con 0x7f33ac1f4410 2024-04-24T23:50:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.792+0000 7f33b25d5640 1 --2- 172.21.15.16:0/1217723907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac1f4410 0x7f33ac0c7de0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f33a400cda0 tx=0x7f33a400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.793+0000 7f33a27fc640 1 -- 172.21.15.16:0/1217723907 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4003b90 con 0x7f33ac1f4410 2024-04-24T23:50:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.793+0000 7f33a27fc640 1 -- 172.21.15.16:0/1217723907 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a4003d30 con 0x7f33ac1f4410 2024-04-24T23:50:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.793+0000 7f33b4038640 1 -- 172.21.15.16:0/1217723907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33ac0af350 con 0x7f33ac1f4410 2024-04-24T23:50:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.793+0000 7f33a27fc640 1 -- 172.21.15.16:0/1217723907 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4004240 con 0x7f33ac1f4410 2024-04-24T23:50:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.793+0000 7f33b4038640 1 -- 172.21.15.16:0/1217723907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33ac0af5f0 con 0x7f33ac1f4410 2024-04-24T23:50:53.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.794+0000 7f33b4038640 1 -- 172.21.15.16:0/1217723907 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f33ac0b4660 con 0x7f33ac1f4410 2024-04-24T23:50:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.794+0000 7f33a27fc640 1 -- 172.21.15.16:0/1217723907 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f33a40043e0 con 0x7f33ac1f4410 2024-04-24T23:50:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.795+0000 7f33a27fc640 1 --2- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f339006be60 0x7f339006e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.795+0000 7f33a27fc640 1 -- 172.21.15.16:0/1217723907 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f33a408b6c0 con 0x7f33ac1f4410 2024-04-24T23:50:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.795+0000 7f33a27fc640 1 --2- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:6824/2447858915,v1:172.21.15.156:6825/2447858915] conn(0x7f33900723a0 0x7f33900747e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.795+0000 7f33a27fc640 1 -- 172.21.15.16:0/1217723907 --> [v2:172.21.15.156:6824/2447858915,v1:172.21.15.156:6825/2447858915] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3390074eb0 con 0x7f33900723a0 2024-04-24T23:50:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.795+0000 7f33a27fc640 1 -- 172.21.15.16:0/1217723907 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7f33a4092050 con 0x7f33ac1f4410 2024-04-24T23:50:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.795+0000 7f33b1dd4640 1 --2- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f339006be60 0x7f339006e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.795+0000 7f33b15d3640 1 --2- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:6824/2447858915,v1:172.21.15.156:6825/2447858915] conn(0x7f33900723a0 0x7f33900747e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.795+0000 7f33b1dd4640 1 --2- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f339006be60 0x7f339006e320 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f33a8034000 tx=0x7f33a802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.796+0000 7f33b15d3640 1 --2- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:6824/2447858915,v1:172.21.15.156:6825/2447858915] conn(0x7f33900723a0 0x7f33900747e0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.796+0000 7f33a27fc640 1 -- 172.21.15.16:0/1217723907 <== osd.7 v2:172.21.15.156:6824/2447858915 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3390074eb0 con 0x7f33900723a0 2024-04-24T23:50:53.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.810+0000 7f33b4038640 1 -- 172.21.15.16:0/1217723907 --> [v2:172.21.15.156:6824/2447858915,v1:172.21.15.156:6825/2447858915] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f33ac068f90 con 0x7f33900723a0 2024-04-24T23:50:53.809 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.810+0000 7f33a27fc640 1 -- 172.21.15.16:0/1217723907 <== osd.7 v2:172.21.15.156:6824/2447858915 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f33ac068f90 con 0x7f33900723a0 2024-04-24T23:50:53.810 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-04-24T23:50:53.810 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-04-24T23:50:53.810 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-04-24T23:50:53.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.810+0000 7f33b4038640 1 -- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:6824/2447858915,v1:172.21.15.156:6825/2447858915] conn(0x7f33900723a0 msgr2=0x7f33900747e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.810+0000 7f33b4038640 1 --2- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:6824/2447858915,v1:172.21.15.156:6825/2447858915] conn(0x7f33900723a0 0x7f33900747e0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.811+0000 7f33b4038640 1 -- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f339006be60 msgr2=0x7f339006e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.811+0000 7f33b4038640 1 --2- 172.21.15.16:0/1217723907 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f339006be60 0x7f339006e320 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f33a8034000 tx=0x7f33a802e000 comp rx=0 tx=0).stop 2024-04-24T23:50:53.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.811+0000 7f33b4038640 1 -- 172.21.15.16:0/1217723907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac1f4410 msgr2=0x7f33ac0c7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.811+0000 7f33b4038640 1 --2- 172.21.15.16:0/1217723907 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f33ac1f4410 0x7f33ac0c7de0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f33a400cda0 tx=0x7f33a400a660 comp rx=0 tx=0).stop 2024-04-24T23:50:53.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.811+0000 7f33b25d5640 1 -- 172.21.15.16:0/1217723907 reap_dead start 2024-04-24T23:50:53.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.811+0000 7f33b4038640 1 -- 172.21.15.16:0/1217723907 shutdown_connections 2024-04-24T23:50:53.811 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.811+0000 7f33b4038640 1 -- 172.21.15.16:0/1217723907 >> 172.21.15.16:0/1217723907 conn(0x7f33ac0d1bc0 msgr2=0x7f33ac0cee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:53.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.811+0000 7f33b4038640 1 -- 172.21.15.16:0/1217723907 shutdown_connections 2024-04-24T23:50:53.812 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.811+0000 7f33b4038640 1 -- 172.21.15.16:0/1217723907 wait complete. 2024-04-24T23:50:53.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-04-24T23:50:53.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.922+0000 7f0d2ce1b640 1 Processor -- start 2024-04-24T23:50:53.922 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.922+0000 7f0d2ce1b640 1 -- start start 2024-04-24T23:50:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d2ce1b640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0d2806cb00 0x7f0d2815c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d2ce1b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d280729a0 0x7f0d280ced40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d2ce1b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d2806cee0 con 0x7f0d280ccc70 2024-04-24T23:50:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d2ce1b640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f0d280ce760 con 0x7f0d2806cb00 2024-04-24T23:50:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d2ce1b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0d2815c8b0 con 0x7f0d280729a0 2024-04-24T23:50:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d2659c640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0d2806cb00 0x7f0d2815c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d1ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0d280ccc70 0x7f0d28072290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:57702/0 (socket says 172.21.15.16:57702) 2024-04-24T23:50:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d1ffff640 1 -- 172.21.15.16:0/3852961424 learned_addr learned my addr 172.21.15.16:0/3852961424 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:50:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d26d9d640 1 --2- 172.21.15.16:0/3852961424 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d280729a0 0x7f0d280ced40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d25d9b640 1 -- 172.21.15.16:0/3852961424 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1204218502 0 0) 0x7f0d2806cee0 con 0x7f0d280ccc70 2024-04-24T23:50:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d25d9b640 1 -- 172.21.15.16:0/3852961424 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d14003520 con 0x7f0d280ccc70 2024-04-24T23:50:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d2659c640 1 -- 172.21.15.16:0/3852961424 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d280729a0 msgr2=0x7f0d280ced40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d2659c640 1 --2- 172.21.15.16:0/3852961424 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d280729a0 0x7f0d280ced40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d2659c640 1 -- 172.21.15.16:0/3852961424 >> v1:172.21.15.16:6789/0 conn(0x7f0d280ccc70 legacy=0x7f0d28072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.923+0000 7f0d2659c640 1 -- 172.21.15.16:0/3852961424 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d2815cb90 con 0x7f0d2806cb00 2024-04-24T23:50:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.924+0000 7f0d2659c640 1 --2- 172.21.15.16:0/3852961424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2806cb00 0x7f0d2815c250 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0d180097a0 tx=0x7f0d1802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0df49c2c34d9285 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.924+0000 7f0d25d9b640 1 -- 172.21.15.16:0/3852961424 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d18037070 con 0x7f0d2806cb00 2024-04-24T23:50:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.924+0000 7f0d25d9b640 1 -- 172.21.15.16:0/3852961424 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d18032440 con 0x7f0d2806cb00 2024-04-24T23:50:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.924+0000 7f0d25d9b640 1 -- 172.21.15.16:0/3852961424 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d18032810 con 0x7f0d2806cb00 2024-04-24T23:50:53.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.925+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3852961424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2806cb00 msgr2=0x7f0d2815c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.925+0000 7f0d2ce1b640 1 --2- 172.21.15.16:0/3852961424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2806cb00 0x7f0d2815c250 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0d180097a0 tx=0x7f0d1802a180 comp rx=0 tx=0).stop 2024-04-24T23:50:53.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.925+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3852961424 shutdown_connections 2024-04-24T23:50:53.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.925+0000 7f0d2ce1b640 1 --2- 172.21.15.16:0/3852961424 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d280729a0 0x7f0d280ced40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.925+0000 7f0d2ce1b640 1 --2- 172.21.15.16:0/3852961424 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d2806cb00 0x7f0d2815c250 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.925+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3852961424 >> 172.21.15.16:0/3852961424 conn(0x7f0d280d0260 msgr2=0x7f0d280d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:53.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.925+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3852961424 shutdown_connections 2024-04-24T23:50:53.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.925+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3852961424 wait complete. 2024-04-24T23:50:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.926+0000 7f0d2ce1b640 1 Processor -- start 2024-04-24T23:50:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.926+0000 7f0d2ce1b640 1 -- start start 2024-04-24T23:50:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d2ce1b640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d2806cb00 0x7f0d28142430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d2ce1b640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d280729a0 0x7f0d2806baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d2ce1b640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d280ccc70 0x7f0d2815afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d2ce1b640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2806b0e0 con 0x7f0d280ccc70 2024-04-24T23:50:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d2ce1b640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0d2815d150 con 0x7f0d2806cb00 2024-04-24T23:50:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d2ce1b640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f0d280cd0f0 con 0x7f0d280729a0 2024-04-24T23:50:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d26d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d280ccc70 0x7f0d2815afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d26d9d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d280ccc70 0x7f0d2815afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49168/0 (socket says 172.21.15.16:49168) 2024-04-24T23:50:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d26d9d640 1 -- 172.21.15.16:0/3501473830 learned_addr learned my addr 172.21.15.16:0/3501473830 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d2659c640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d2806cb00 0x7f0d28142430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d1ffff640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d280729a0 0x7f0d2806baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d26d9d640 1 -- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d280729a0 msgr2=0x7f0d2806baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d26d9d640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d280729a0 0x7f0d2806baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d26d9d640 1 -- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d2806cb00 msgr2=0x7f0d28142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d26d9d640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d2806cb00 0x7f0d28142430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d26d9d640 1 -- 172.21.15.16:0/3501473830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d28142a10 con 0x7f0d280ccc70 2024-04-24T23:50:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d1ffff640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d280729a0 0x7f0d2806baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:50:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.927+0000 7f0d26d9d640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d280ccc70 0x7f0d2815afd0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0d2000efc0 tx=0x7f0d2000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.928+0000 7f0d1effd640 1 -- 172.21.15.16:0/3501473830 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d20018020 con 0x7f0d280ccc70 2024-04-24T23:50:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.928+0000 7f0d1effd640 1 -- 172.21.15.16:0/3501473830 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d20014480 con 0x7f0d280ccc70 2024-04-24T23:50:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.928+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3501473830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d28142cf0 con 0x7f0d280ccc70 2024-04-24T23:50:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.928+0000 7f0d1effd640 1 -- 172.21.15.16:0/3501473830 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d20014870 con 0x7f0d280ccc70 2024-04-24T23:50:53.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.928+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3501473830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d2806bfe0 con 0x7f0d280ccc70 2024-04-24T23:50:53.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.929+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3501473830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d2804f5e0 con 0x7f0d280ccc70 2024-04-24T23:50:53.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.929+0000 7f0d1effd640 1 -- 172.21.15.16:0/3501473830 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0d20005d10 con 0x7f0d280ccc70 2024-04-24T23:50:53.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.929+0000 7f0d1effd640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d0806be60 0x7f0d0806e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:53.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.930+0000 7f0d1effd640 1 -- 172.21.15.16:0/3501473830 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f0d2008b930 con 0x7f0d280ccc70 2024-04-24T23:50:53.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.930+0000 7f0d2659c640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d0806be60 0x7f0d0806e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:53.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.930+0000 7f0d2659c640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d0806be60 0x7f0d0806e320 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0d18034000 tx=0x7f0d1802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:53.936 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:53.934+0000 7f0d1effd640 1 -- 172.21.15.16:0/3501473830 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f0d20022070 con 0x7f0d280ccc70 2024-04-24T23:50:54.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.116+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3501473830 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f0d28072e90 con 0x7f0d0806be60 2024-04-24T23:50:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.117+0000 7f0d1effd640 1 -- 172.21.15.16:0/3501473830 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f0d28072e90 con 0x7f0d0806be60 2024-04-24T23:50:54.117 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 7 to deep-scrub 2024-04-24T23:50:54.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.119+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d0806be60 msgr2=0x7f0d0806e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:54.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.119+0000 7f0d2ce1b640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d0806be60 0x7f0d0806e320 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0d18034000 tx=0x7f0d1802e000 comp rx=0 tx=0).stop 2024-04-24T23:50:54.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.119+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3501473830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d280ccc70 msgr2=0x7f0d2815afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:54.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.119+0000 7f0d2ce1b640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d280ccc70 0x7f0d2815afd0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f0d2000efc0 tx=0x7f0d2000c490 comp rx=0 tx=0).stop 2024-04-24T23:50:54.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.119+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3501473830 shutdown_connections 2024-04-24T23:50:54.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.119+0000 7f0d2ce1b640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f0d0806be60 0x7f0d0806e320 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.119+0000 7f0d2ce1b640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d280ccc70 0x7f0d2815afd0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.119+0000 7f0d2ce1b640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f0d280729a0 0x7f0d2806baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.119+0000 7f0d2ce1b640 1 --2- 172.21.15.16:0/3501473830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0d2806cb00 0x7f0d28142430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.119+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3501473830 >> 172.21.15.16:0/3501473830 conn(0x7f0d280d0260 msgr2=0x7f0d280b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:54.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.119+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3501473830 shutdown_connections 2024-04-24T23:50:54.120 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.119+0000 7f0d2ce1b640 1 -- 172.21.15.16:0/3501473830 wait complete. 2024-04-24T23:50:54.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:50:54.179 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:50:54.182 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:50:54.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.231+0000 7fb056a93640 1 Processor -- start 2024-04-24T23:50:54.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.231+0000 7fb056a93640 1 -- start start 2024-04-24T23:50:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb056a93640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb05015e020 0x7fb0500cf270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb056a93640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0500b2bb0 0x7fb0500cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb056a93640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fb0500cfcf0 0x7fb050161bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb056a93640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7fb0500b3390 con 0x7fb0500b2bb0 2024-04-24T23:50:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb056a93640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fb05006bfc0 con 0x7fb0500cfcf0 2024-04-24T23:50:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb056a93640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb05006c2a0 con 0x7fb05015e020 2024-04-24T23:50:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb04ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0500b2bb0 0x7fb0500cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb04ffff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0500b2bb0 0x7fb0500cf7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49180/0 (socket says 172.21.15.16:49180) 2024-04-24T23:50:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb04ffff640 1 -- 172.21.15.16:0/544506544 learned_addr learned my addr 172.21.15.16:0/544506544 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb05482f640 1 --2- 172.21.15.16:0/544506544 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb05015e020 0x7fb0500cf270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb055030640 1 --2- 172.21.15.16:0/544506544 >> v2:172.21.15.156:3300/0 conn(0x7fb0500cfcf0 0x7fb050161bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb04ffff640 1 -- 172.21.15.16:0/544506544 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb05015e020 msgr2=0x7fb0500cf270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb04ffff640 1 --2- 172.21.15.16:0/544506544 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb05015e020 0x7fb0500cf270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb04ffff640 1 -- 172.21.15.16:0/544506544 >> v2:172.21.15.156:3300/0 conn(0x7fb0500cfcf0 msgr2=0x7fb050161bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:50:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb04ffff640 1 --2- 172.21.15.16:0/544506544 >> v2:172.21.15.156:3300/0 conn(0x7fb0500cfcf0 0x7fb050161bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.232+0000 7fb04ffff640 1 -- 172.21.15.16:0/544506544 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb05006c580 con 0x7fb0500b2bb0 2024-04-24T23:50:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.233+0000 7fb04ffff640 1 --2- 172.21.15.16:0/544506544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0500b2bb0 0x7fb0500cf7b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb05006b220 tx=0x7fb03c02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d5f730bb5a9bcf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.233+0000 7fb04f7fe640 1 -- 172.21.15.16:0/544506544 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c037070 con 0x7fb0500b2bb0 2024-04-24T23:50:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.233+0000 7fb04f7fe640 1 -- 172.21.15.16:0/544506544 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb03c032440 con 0x7fb0500b2bb0 2024-04-24T23:50:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.233+0000 7fb04f7fe640 1 -- 172.21.15.16:0/544506544 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c032810 con 0x7fb0500b2bb0 2024-04-24T23:50:54.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.234+0000 7fb056a93640 1 -- 172.21.15.16:0/544506544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0500b2bb0 msgr2=0x7fb0500cf7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:54.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.234+0000 7fb056a93640 1 --2- 172.21.15.16:0/544506544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0500b2bb0 0x7fb0500cf7b0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb05006b220 tx=0x7fb03c02a140 comp rx=0 tx=0).stop 2024-04-24T23:50:54.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.234+0000 7fb056a93640 1 -- 172.21.15.16:0/544506544 shutdown_connections 2024-04-24T23:50:54.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.234+0000 7fb056a93640 1 --2- 172.21.15.16:0/544506544 >> v2:172.21.15.156:3300/0 conn(0x7fb0500cfcf0 0x7fb050161bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.234+0000 7fb056a93640 1 --2- 172.21.15.16:0/544506544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0500b2bb0 0x7fb0500cf7b0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.234+0000 7fb056a93640 1 --2- 172.21.15.16:0/544506544 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb05015e020 0x7fb0500cf270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.234+0000 7fb056a93640 1 -- 172.21.15.16:0/544506544 >> 172.21.15.16:0/544506544 conn(0x7fb0500d3e10 msgr2=0x7fb0500d4220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:54.236 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.234+0000 7fb056a93640 1 -- 172.21.15.16:0/544506544 shutdown_connections 2024-04-24T23:50:54.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.234+0000 7fb056a93640 1 -- 172.21.15.16:0/544506544 wait complete. 2024-04-24T23:50:54.239 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.235+0000 7fb056a93640 1 Processor -- start 2024-04-24T23:50:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.235+0000 7fb056a93640 1 -- start start 2024-04-24T23:50:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.236+0000 7fb056a93640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0500b2bb0 0x7fb05014e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.236+0000 7fb056a93640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0500cfcf0 0x7fb0500ca140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.236+0000 7fb056a93640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb05015e020 0x7fb0500b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.236+0000 7fb056a93640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fb05015e400 con 0x7fb0500cfcf0 2024-04-24T23:50:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.236+0000 7fb056a93640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb0500d1a80 con 0x7fb05015e020 2024-04-24T23:50:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.236+0000 7fb056a93640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fb050162130 con 0x7fb0500b2bb0 2024-04-24T23:50:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.236+0000 7fb04ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0500cfcf0 0x7fb0500ca140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.236+0000 7fb04ffff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0500cfcf0 0x7fb0500ca140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49182/0 (socket says 172.21.15.16:49182) 2024-04-24T23:50:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.236+0000 7fb04ffff640 1 -- 172.21.15.16:0/1704146062 learned_addr learned my addr 172.21.15.16:0/1704146062 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:50:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.236+0000 7fb05482f640 1 --2- 172.21.15.16:0/1704146062 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0500b2bb0 0x7fb05014e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.237+0000 7fb055030640 1 --2- 172.21.15.16:0/1704146062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb05015e020 0x7fb0500b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:54.240 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.237+0000 7fb04ffff640 1 -- 172.21.15.16:0/1704146062 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0500b2bb0 msgr2=0x7fb05014e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.237+0000 7fb04ffff640 1 --2- 172.21.15.16:0/1704146062 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0500b2bb0 0x7fb05014e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.237+0000 7fb04ffff640 1 -- 172.21.15.16:0/1704146062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb05015e020 msgr2=0x7fb0500b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.237+0000 7fb04ffff640 1 --2- 172.21.15.16:0/1704146062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb05015e020 0x7fb0500b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.237+0000 7fb04ffff640 1 -- 172.21.15.16:0/1704146062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0500c90a0 con 0x7fb0500cfcf0 2024-04-24T23:50:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.237+0000 7fb04ffff640 1 --2- 172.21.15.16:0/1704146062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0500cfcf0 0x7fb0500ca140 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb03c02a180 tx=0x7fb03c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.237+0000 7fb04d7fa640 1 -- 172.21.15.16:0/1704146062 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c041020 con 0x7fb0500cfcf0 2024-04-24T23:50:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.237+0000 7fb04d7fa640 1 -- 172.21.15.16:0/1704146062 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb03c03d620 con 0x7fb0500cfcf0 2024-04-24T23:50:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.237+0000 7fb056a93640 1 -- 172.21.15.16:0/1704146062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0500c9380 con 0x7fb0500cfcf0 2024-04-24T23:50:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.237+0000 7fb04d7fa640 1 -- 172.21.15.16:0/1704146062 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb03c037040 con 0x7fb0500cfcf0 2024-04-24T23:50:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.237+0000 7fb056a93640 1 -- 172.21.15.16:0/1704146062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0500c9c20 con 0x7fb0500cfcf0 2024-04-24T23:50:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.238+0000 7fb056a93640 1 -- 172.21.15.16:0/1704146062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0500c9770 con 0x7fb0500cfcf0 2024-04-24T23:50:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.239+0000 7fb04d7fa640 1 -- 172.21.15.16:0/1704146062 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb03c044020 con 0x7fb0500cfcf0 2024-04-24T23:50:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.239+0000 7fb04d7fa640 1 --2- 172.21.15.16:0/1704146062 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb03406be30 0x7fb03406e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:50:54.241 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.239+0000 7fb04d7fa640 1 -- 172.21.15.16:0/1704146062 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7fb03c0ab350 con 0x7fb0500cfcf0 2024-04-24T23:50:54.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.239+0000 7fb05482f640 1 --2- 172.21.15.16:0/1704146062 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb03406be30 0x7fb03406e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:50:54.242 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.240+0000 7fb05482f640 1 --2- 172.21.15.16:0/1704146062 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb03406be30 0x7fb03406e2f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb044009aa0 tx=0x7fb044009190 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:50:54.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.244+0000 7fb04d7fa640 1 -- 172.21.15.16:0/1704146062 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fb03c04b030 con 0x7fb0500cfcf0 2024-04-24T23:50:54.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.425+0000 7fb056a93640 1 -- 172.21.15.16:0/1704146062 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb05006dfe0 con 0x7fb03406be30 2024-04-24T23:50:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.426+0000 7fb04d7fa640 1 -- 172.21.15.16:0/1704146062 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13372 (secure 0 0 0) 0x7fb05006dfe0 con 0x7fb03406be30 2024-04-24T23:50:54.426 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:50:54.426 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-24T23:50:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.429+0000 7fb056a93640 1 -- 172.21.15.16:0/1704146062 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb03406be30 msgr2=0x7fb03406e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.429+0000 7fb056a93640 1 --2- 172.21.15.16:0/1704146062 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb03406be30 0x7fb03406e2f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb044009aa0 tx=0x7fb044009190 comp rx=0 tx=0).stop 2024-04-24T23:50:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.429+0000 7fb056a93640 1 -- 172.21.15.16:0/1704146062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0500cfcf0 msgr2=0x7fb0500ca140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:50:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.429+0000 7fb056a93640 1 --2- 172.21.15.16:0/1704146062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0500cfcf0 0x7fb0500ca140 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb03c02a180 tx=0x7fb03c0046c0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.429+0000 7fb056a93640 1 -- 172.21.15.16:0/1704146062 shutdown_connections 2024-04-24T23:50:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.429+0000 7fb056a93640 1 --2- 172.21.15.16:0/1704146062 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fb03406be30 0x7fb03406e2f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.429+0000 7fb056a93640 1 --2- 172.21.15.16:0/1704146062 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb05015e020 0x7fb0500b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.429+0000 7fb056a93640 1 --2- 172.21.15.16:0/1704146062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0500cfcf0 0x7fb0500ca140 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.429+0000 7fb056a93640 1 --2- 172.21.15.16:0/1704146062 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fb0500b2bb0 0x7fb05014e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:50:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.429+0000 7fb056a93640 1 -- 172.21.15.16:0/1704146062 >> 172.21.15.16:0/1704146062 conn(0x7fb0500d3e10 msgr2=0x7fb05015d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:50:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.429+0000 7fb056a93640 1 -- 172.21.15.16:0/1704146062 shutdown_connections 2024-04-24T23:50:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:50:54.429+0000 7fb056a93640 1 -- 172.21.15.16:0/1704146062 wait complete. 2024-04-24T23:50:54.439 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-04-24T23:50:53.148288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":254868,"kb_used_data":5416,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719852,"statfs":{"total":773094113280,"available":772833128448,"internally_reserved":0,"allocated":5545984,"data_stored":3103904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8949,"num_objects":-23,"num_object_clones":0,"num_object_copies":-46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":0,"num_read":-80,"num_read_kb":-88,"num_write":-61,"num_write_kb":-28,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001918"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":221,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T23:50:46.582781+0000","last_change":"2024-04-24T23:48:45.082459+0000","last_active":"2024-04-24T23:50:46.582781+0000","last_peered":"2024-04-24T23:50:46.582781+0000","last_clean":"2024-04-24T23:50:46.582781+0000","last_became_active":"2024-04-24T23:48:45.081495+0000","last_became_peered":"2024-04-24T23:48:45.081495+0000","last_unstale":"2024-04-24T23:50:46.582781+0000","last_undegraded":"2024-04-24T23:50:46.582781+0000","last_fullsized":"2024-04-24T23:50:46.582781+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T23:48:43.440993+0000","last_clean_scrub_stamp":"2024-04-24T23:48:43.440993+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T09:53:20.832511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32296,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339544,"statfs":{"total":96636764160,"available":96603693056,"internally_reserved":0,"allocated":1138688,"data_stored":830764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31716,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574878,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31716,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574877,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32292,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339548,"statfs":{"total":96636764160,"available":96603697152,"internally_reserved":0,"allocated":1138688,"data_stored":830764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:50:54.442 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-04-24T23:48:43.440993+0000 time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=48, tm_sec=43, tm_wday=2, tm_yday=115, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=4, tm_mday=24, tm_hour=23, tm_min=50, tm_sec=49, tm_wday=2, tm_yday=115, tm_isdst=0) 2024-04-24T23:50:54.442 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-04-24T23:51:14.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T23:51:14.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.558+0000 7fa0f8790640 1 Processor -- start 2024-04-24T23:51:14.558 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.558+0000 7fa0f8790640 1 -- start start 2024-04-24T23:51:14.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.559+0000 7fa0f8790640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7fa0f0162380 0x7fa0f0168f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.559+0000 7fa0f8790640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f0162790 con 0x7fa0f00729a0 2024-04-24T23:51:14.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.559+0000 7fa0f8790640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0f00b4c00 con 0x7fa0f015df40 2024-04-24T23:51:14.559 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.559+0000 7fa0f8790640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7fa0f00ae2b0 con 0x7fa0f0162380 2024-04-24T23:51:14.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f5d2b640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa0f00729a0 0x7fa0f0161c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34768/0 (socket says 172.21.15.16:34768) 2024-04-24T23:51:14.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f5d2b640 1 -- 172.21.15.16:0/3265753551 learned_addr learned my addr 172.21.15.16:0/3265753551 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:51:14.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f6d2d640 1 --2- 172.21.15.16:0/3265753551 >> v2:172.21.15.156:3300/0 conn(0x7fa0f0162380 0x7fa0f0168f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 -- 172.21.15.16:0/3265753551 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 749995314 0 0) 0x7fa0e4001110 con 0x7fa0f00729a0 2024-04-24T23:51:14.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 -- 172.21.15.16:0/3265753551 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0d8003520 con 0x7fa0f00729a0 2024-04-24T23:51:14.562 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 -- 172.21.15.16:0/3265753551 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3344483092 0 0) 0x7fa0d8003520 con 0x7fa0f00729a0 2024-04-24T23:51:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 -- 172.21.15.16:0/3265753551 >> v2:172.21.15.156:3300/0 conn(0x7fa0f0162380 msgr2=0x7fa0f0168f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:51:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 --2- 172.21.15.16:0/3265753551 >> v2:172.21.15.156:3300/0 conn(0x7fa0f0162380 0x7fa0f0168f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 -- 172.21.15.16:0/3265753551 >> v1:172.21.15.156:6789/0 conn(0x7fa0f015df40 legacy=0x7fa0f0072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 -- 172.21.15.16:0/3265753551 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f0162a10 con 0x7fa0f00729a0 2024-04-24T23:51:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 -- 172.21.15.16:0/3265753551 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7fa0e4001840 con 0x7fa0f00729a0 2024-04-24T23:51:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 -- 172.21.15.16:0/3265753551 >> v1:172.21.15.16:6789/0 conn(0x7fa0f00729a0 legacy=0x7fa0f0161c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 --2- 172.21.15.16:0/3265753551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0d80039b0 0x7fa0d8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 --2- 172.21.15.16:0/3265753551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0d8004800 0x7fa0d8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 --2- 172.21.15.16:0/3265753551 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0d80051c0 0x7fa0d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 -- 172.21.15.16:0/3265753551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f00ae2b0 con 0x7fa0d80039b0 2024-04-24T23:51:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 -- 172.21.15.16:0/3265753551 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa0d800c060 con 0x7fa0d8004800 2024-04-24T23:51:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f552a640 1 -- 172.21.15.16:0/3265753551 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fa0d800c260 con 0x7fa0d80051c0 2024-04-24T23:51:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.560+0000 7fa0f652c640 1 --2- 172.21.15.16:0/3265753551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0d80039b0 0x7fa0d8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.563 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.561+0000 7fa0f5d2b640 1 --2- 172.21.15.16:0/3265753551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0d8004800 0x7fa0d8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.561+0000 7fa0f6d2d640 1 --2- 172.21.15.16:0/3265753551 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0d80051c0 0x7fa0d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.561+0000 7fa0f652c640 1 -- 172.21.15.16:0/3265753551 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0d80051c0 msgr2=0x7fa0d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.561+0000 7fa0f652c640 1 --2- 172.21.15.16:0/3265753551 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0d80051c0 0x7fa0d800b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.561+0000 7fa0f652c640 1 -- 172.21.15.16:0/3265753551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0d8004800 msgr2=0x7fa0d8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.561+0000 7fa0f652c640 1 --2- 172.21.15.16:0/3265753551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0d8004800 0x7fa0d8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.561+0000 7fa0f652c640 1 -- 172.21.15.16:0/3265753551 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0d8003520 con 0x7fa0d80039b0 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.561+0000 7fa0f6d2d640 1 -- 172.21.15.16:0/3265753551 reap_dead start 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.562+0000 7fa0f652c640 1 --2- 172.21.15.16:0/3265753551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0d80039b0 0x7fa0d8003dd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8001750 tx=0x7fa0d8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94617ce047175e24 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.562+0000 7fa0f552a640 1 -- 172.21.15.16:0/3265753551 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8001370 con 0x7fa0d80039b0 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.562+0000 7fa0f552a640 1 -- 172.21.15.16:0/3265753551 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0e80052d0 con 0x7fa0d80039b0 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.562+0000 7fa0f552a640 1 -- 172.21.15.16:0/3265753551 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e802abc0 con 0x7fa0d80039b0 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.562+0000 7fa0f8790640 1 -- 172.21.15.16:0/3265753551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0d80039b0 msgr2=0x7fa0d8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.562+0000 7fa0f8790640 1 --2- 172.21.15.16:0/3265753551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0d80039b0 0x7fa0d8003dd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8001750 tx=0x7fa0d8005830 comp rx=0 tx=0).stop 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.562+0000 7fa0f8790640 1 -- 172.21.15.16:0/3265753551 shutdown_connections 2024-04-24T23:51:14.564 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.562+0000 7fa0f8790640 1 --2- 172.21.15.16:0/3265753551 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0d80039b0 0x7fa0d8003dd0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.562+0000 7fa0f8790640 1 -- 172.21.15.16:0/3265753551 >> 172.21.15.16:0/3265753551 conn(0x7fa0f00d0260 msgr2=0x7fa0f00d0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:51:14.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.562+0000 7fa0f8790640 1 -- 172.21.15.16:0/3265753551 shutdown_connections 2024-04-24T23:51:14.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.562+0000 7fa0f8790640 1 -- 172.21.15.16:0/3265753551 wait complete. 2024-04-24T23:51:14.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.564+0000 7fa0f8790640 1 Processor -- start 2024-04-24T23:51:14.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.564+0000 7fa0f8790640 1 -- start start 2024-04-24T23:51:14.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.564+0000 7fa0f8790640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0d80039b0 0x7fa0f0157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.567 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.564+0000 7fa0f8790640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0f0154050 0x7fa0f006be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.564+0000 7fa0f8790640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f006d110 0x7fa0f015e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.564+0000 7fa0f8790640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f00adf40 con 0x7fa0f006d110 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.564+0000 7fa0f8790640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa0f00adc60 con 0x7fa0f0154050 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.564+0000 7fa0f8790640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7fa0f00ae740 con 0x7fa0d80039b0 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.564+0000 7fa0f6d2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f006d110 0x7fa0f015e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0f6d2d640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f006d110 0x7fa0f015e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58626/0 (socket says 172.21.15.16:58626) 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0f6d2d640 1 -- 172.21.15.16:0/1817013652 learned_addr learned my addr 172.21.15.16:0/1817013652 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0f5d2b640 1 --2- 172.21.15.16:0/1817013652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0f0154050 0x7fa0f006be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0f652c640 1 --2- 172.21.15.16:0/1817013652 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0d80039b0 0x7fa0f0157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0f6d2d640 1 -- 172.21.15.16:0/1817013652 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0d80039b0 msgr2=0x7fa0f0157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0f6d2d640 1 --2- 172.21.15.16:0/1817013652 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0d80039b0 0x7fa0f0157510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0f6d2d640 1 -- 172.21.15.16:0/1817013652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0f0154050 msgr2=0x7fa0f006be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0f6d2d640 1 --2- 172.21.15.16:0/1817013652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0f0154050 0x7fa0f006be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0f6d2d640 1 -- 172.21.15.16:0/1817013652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f006c390 con 0x7fa0f006d110 2024-04-24T23:51:14.568 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0f6d2d640 1 --2- 172.21.15.16:0/1817013652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f006d110 0x7fa0f015e320 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00b3200 tx=0x7fa0e000f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:51:14.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0eeffd640 1 -- 172.21.15.16:0/1817013652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e000fce0 con 0x7fa0f006d110 2024-04-24T23:51:14.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0eeffd640 1 -- 172.21.15.16:0/1817013652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0e000fe80 con 0x7fa0f006d110 2024-04-24T23:51:14.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0eeffd640 1 -- 172.21.15.16:0/1817013652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e0015670 con 0x7fa0f006d110 2024-04-24T23:51:14.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0f8790640 1 -- 172.21.15.16:0/1817013652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0f0157af0 con 0x7fa0f006d110 2024-04-24T23:51:14.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.565+0000 7fa0f8790640 1 -- 172.21.15.16:0/1817013652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0f006c910 con 0x7fa0f006d110 2024-04-24T23:51:14.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.566+0000 7fa0f8790640 1 -- 172.21.15.16:0/1817013652 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0f006e5c0 con 0x7fa0f006d110 2024-04-24T23:51:14.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.567+0000 7fa0eeffd640 1 -- 172.21.15.16:0/1817013652 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa0e0015810 con 0x7fa0f006d110 2024-04-24T23:51:14.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.567+0000 7fa0eeffd640 1 --2- 172.21.15.16:0/1817013652 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa0dc06be60 0x7fa0dc06e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.567+0000 7fa0eeffd640 1 -- 172.21.15.16:0/1817013652 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7fa0e008b600 con 0x7fa0f006d110 2024-04-24T23:51:14.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.567+0000 7fa0f652c640 1 --2- 172.21.15.16:0/1817013652 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa0dc06be60 0x7fa0dc06e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.568+0000 7fa0f652c640 1 --2- 172.21.15.16:0/1817013652 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa0dc06be60 0x7fa0dc06e320 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8032000 tx=0x7fa0e802e000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:51:14.571 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.572+0000 7fa0eeffd640 1 -- 172.21.15.16:0/1817013652 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7fa0e0024030 con 0x7fa0f006d110 2024-04-24T23:51:14.752 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.752+0000 7fa0f8790640 1 -- 172.21.15.16:0/1817013652 --> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa0f00d0040 con 0x7fa0dc06be60 2024-04-24T23:51:14.753 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.754+0000 7fa0eeffd640 1 -- 172.21.15.16:0/1817013652 <== mgr.4107 v2:172.21.15.156:6832/3478836820 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13359 (secure 0 0 0) 0x7fa0f00d0040 con 0x7fa0dc06be60 2024-04-24T23:51:14.753 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:51:14.753 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-04-24T23:51:14.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.755+0000 7fa0f8790640 1 -- 172.21.15.16:0/1817013652 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa0dc06be60 msgr2=0x7fa0dc06e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:14.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.755+0000 7fa0f8790640 1 --2- 172.21.15.16:0/1817013652 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa0dc06be60 0x7fa0dc06e320 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8032000 tx=0x7fa0e802e000 comp rx=0 tx=0).stop 2024-04-24T23:51:14.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.755+0000 7fa0f8790640 1 -- 172.21.15.16:0/1817013652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f006d110 msgr2=0x7fa0f015e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:14.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.755+0000 7fa0f8790640 1 --2- 172.21.15.16:0/1817013652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f006d110 0x7fa0f015e320 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fa0f00b3200 tx=0x7fa0e000f1b0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.755+0000 7fa0f8790640 1 -- 172.21.15.16:0/1817013652 shutdown_connections 2024-04-24T23:51:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.755+0000 7fa0f8790640 1 --2- 172.21.15.16:0/1817013652 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7fa0dc06be60 0x7fa0dc06e320 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.755+0000 7fa0f8790640 1 --2- 172.21.15.16:0/1817013652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa0f006d110 0x7fa0f015e320 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.755+0000 7fa0f8790640 1 --2- 172.21.15.16:0/1817013652 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0f0154050 0x7fa0f006be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.755+0000 7fa0f8790640 1 --2- 172.21.15.16:0/1817013652 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7fa0d80039b0 0x7fa0f0157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.755+0000 7fa0f8790640 1 -- 172.21.15.16:0/1817013652 >> 172.21.15.16:0/1817013652 conn(0x7fa0f00d0260 msgr2=0x7fa0f0156140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:51:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.756+0000 7fa0f8790640 1 -- 172.21.15.16:0/1817013652 shutdown_connections 2024-04-24T23:51:14.756 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.756+0000 7fa0f8790640 1 -- 172.21.15.16:0/1817013652 wait complete. 2024-04-24T23:51:14.766 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-04-24T23:51:13.151980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":254856,"kb_used_data":5416,"kb_used_omap":12,"kb_used_meta":249331,"kb_avail":754719864,"statfs":{"total":773094113280,"available":772833140736,"internally_reserved":0,"allocated":5545984,"data_stored":3103904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":255315504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002059"},"pg_stats":[{"pgid":"1.0","version":"46'119","reported_seq":232,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-04-24T23:50:54.721393+0000","last_change":"2024-04-24T23:50:54.721432+0000","last_active":"2024-04-24T23:50:54.721393+0000","last_peered":"2024-04-24T23:50:54.721393+0000","last_clean":"2024-04-24T23:50:54.721393+0000","last_became_active":"2024-04-24T23:48:45.081495+0000","last_became_peered":"2024-04-24T23:48:45.081495+0000","last_unstale":"2024-04-24T23:50:54.721393+0000","last_undegraded":"2024-04-24T23:50:54.721393+0000","last_fullsized":"2024-04-24T23:50:54.721393+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"46'119","last_scrub_stamp":"2024-04-24T23:50:54.721307+0000","last_deep_scrub":"46'119","last_deep_scrub_stamp":"2024-04-24T23:50:54.721307+0000","last_clean_scrub_stamp":"2024-04-24T23:50:54.721307+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-04-26T02:49:06.764424+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1138688,"data_stored":830764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31716,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574882,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31716,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31712,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":544768,"data_stored":240396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1138688,"data_stored":830764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T23:51:14.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-04-24T23:51:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.868+0000 7f9cc1b4f640 1 Processor -- start 2024-04-24T23:51:14.868 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.868+0000 7f9cc1b4f640 1 -- start start 2024-04-24T23:51:14.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cc1b4f640 1 --2- >> v2:172.21.15.156:3300/0 conn(0x7f9cbc0b5270 0x7f9cbc15d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cc1b4f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc0cdb80 0x7f9cbc06e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.872 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cc1b4f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cbc0cdff0 con 0x7f9cbc15c1e0 2024-04-24T23:51:14.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cc1b4f640 1 -- --> v2:172.21.15.156:3300/0 -- mon_getmap magic: 0 -- 0x7f9cbc15c7b0 con 0x7f9cbc0b5270 2024-04-24T23:51:14.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cc1b4f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f9cbc15ca90 con 0x7f9cbc0cdb80 2024-04-24T23:51:14.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cbaffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9cbc15c1e0 0x7f9cbc0b5650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34780/0 (socket says 172.21.15.16:34780) 2024-04-24T23:51:14.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cbaffd640 1 -- 172.21.15.16:0/146842717 learned_addr learned my addr 172.21.15.16:0/146842717 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-04-24T23:51:14.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cba7fc640 1 -- 172.21.15.16:0/146842717 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3567335002 0 0) 0x7f9cbc0cdff0 con 0x7f9cbc15c1e0 2024-04-24T23:51:14.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cba7fc640 1 -- 172.21.15.16:0/146842717 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ca4003520 con 0x7f9cbc15c1e0 2024-04-24T23:51:14.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cbbfff640 1 --2- 172.21.15.16:0/146842717 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc0cdb80 0x7f9cbc06e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cbb7fe640 1 --2- 172.21.15.16:0/146842717 >> v2:172.21.15.156:3300/0 conn(0x7f9cbc0b5270 0x7f9cbc15d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cba7fc640 1 -- 172.21.15.16:0/146842717 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2937894045 0 0) 0x7f9ca4003520 con 0x7f9cbc15c1e0 2024-04-24T23:51:14.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cba7fc640 1 -- 172.21.15.16:0/146842717 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc0cdb80 msgr2=0x7f9cbc06e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:51:14.873 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cba7fc640 1 --2- 172.21.15.16:0/146842717 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc0cdb80 0x7f9cbc06e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cba7fc640 1 -- 172.21.15.16:0/146842717 >> v2:172.21.15.156:3300/0 conn(0x7f9cbc0b5270 msgr2=0x7f9cbc15d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:51:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cba7fc640 1 --2- 172.21.15.16:0/146842717 >> v2:172.21.15.156:3300/0 conn(0x7f9cbc0b5270 0x7f9cbc15d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cba7fc640 1 -- 172.21.15.16:0/146842717 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cbc15cd70 con 0x7f9cbc15c1e0 2024-04-24T23:51:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cba7fc640 1 -- 172.21.15.16:0/146842717 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2131361546 0 0) 0x7f9cb0001690 con 0x7f9cbc15c1e0 2024-04-24T23:51:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.869+0000 7f9cba7fc640 1 -- 172.21.15.16:0/146842717 >> v1:172.21.15.16:6789/0 conn(0x7f9cbc15c1e0 legacy=0x7f9cbc0b5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cba7fc640 1 --2- 172.21.15.16:0/146842717 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc0cdb80 0x7f9ca40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cba7fc640 1 --2- 172.21.15.16:0/146842717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ca4003f30 0x7f9ca4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cba7fc640 1 --2- 172.21.15.16:0/146842717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca4004d40 0x7f9ca40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cba7fc640 1 -- 172.21.15.16:0/146842717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc15c7b0 con 0x7f9ca4004d40 2024-04-24T23:51:14.874 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cba7fc640 1 -- 172.21.15.16:0/146842717 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc15ca90 con 0x7f9ca4003f30 2024-04-24T23:51:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cba7fc640 1 -- 172.21.15.16:0/146842717 --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f9ca400bac0 con 0x7f9cbc0cdb80 2024-04-24T23:51:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cbb7fe640 1 --2- 172.21.15.16:0/146842717 >> v2:172.21.15.156:3300/0 conn(0x7f9cbc0b5270 0x7f9cbc15d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:51:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cbbfff640 1 --2- 172.21.15.16:0/146842717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca4004d40 0x7f9ca40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cbaffd640 1 --2- 172.21.15.16:0/146842717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ca4003f30 0x7f9ca4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cbb7fe640 1 --2- 172.21.15.16:0/146842717 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc0cdb80 0x7f9ca40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cbbfff640 1 -- 172.21.15.16:0/146842717 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc0cdb80 msgr2=0x7f9ca40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:51:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cbbfff640 1 --2- 172.21.15.16:0/146842717 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc0cdb80 0x7f9ca40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cbbfff640 1 -- 172.21.15.16:0/146842717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ca4003f30 msgr2=0x7f9ca4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:51:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cbbfff640 1 --2- 172.21.15.16:0/146842717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ca4003f30 0x7f9ca4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cbbfff640 1 -- 172.21.15.16:0/146842717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca4003520 con 0x7f9ca4004d40 2024-04-24T23:51:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.870+0000 7f9cbaffd640 1 --2- 172.21.15.16:0/146842717 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9ca4003f30 0x7f9ca4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T23:51:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.871+0000 7f9cbbfff640 1 --2- 172.21.15.16:0/146842717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca4004d40 0x7f9ca40051c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400e170 tx=0x7f9ca4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261763437a8ee662 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:51:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.871+0000 7f9cba7fc640 1 -- 172.21.15.16:0/146842717 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb4034020 con 0x7f9ca4004d40 2024-04-24T23:51:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.871+0000 7f9cba7fc640 1 -- 172.21.15.16:0/146842717 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cb40305f0 con 0x7f9ca4004d40 2024-04-24T23:51:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.871+0000 7f9cba7fc640 1 -- 172.21.15.16:0/146842717 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb4009c10 con 0x7f9ca4004d40 2024-04-24T23:51:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.871+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/146842717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca4004d40 msgr2=0x7f9ca40051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.871+0000 7f9cc1b4f640 1 --2- 172.21.15.16:0/146842717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ca4004d40 0x7f9ca40051c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400e170 tx=0x7f9ca4005830 comp rx=0 tx=0).stop 2024-04-24T23:51:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.871+0000 7f9cbbfff640 1 -- 172.21.15.16:0/146842717 reap_dead start 2024-04-24T23:51:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.871+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/146842717 shutdown_connections 2024-04-24T23:51:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.871+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/146842717 >> 172.21.15.16:0/146842717 conn(0x7f9cbc06c200 msgr2=0x7f9cbc06c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:51:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.871+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/146842717 shutdown_connections 2024-04-24T23:51:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.871+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/146842717 wait complete. 2024-04-24T23:51:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.873+0000 7f9cc1b4f640 1 Processor -- start 2024-04-24T23:51:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.873+0000 7f9cc1b4f640 1 -- start start 2024-04-24T23:51:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.873+0000 7f9cc1b4f640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cbc148a40 0x7f9cbc0b1440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.873+0000 7f9cc1b4f640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc0aec20 0x7f9cbc0b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.873+0000 7f9cc1b4f640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc0b35a0 0x7f9cbc149040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.873+0000 7f9cc1b4f640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc0cf740 con 0x7f9cbc0b35a0 2024-04-24T23:51:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.873+0000 7f9cc1b4f640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9cbc156640 con 0x7f9cbc148a40 2024-04-24T23:51:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.873+0000 7f9cc1b4f640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f9cbc0cf8c0 con 0x7f9cbc0aec20 2024-04-24T23:51:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9cbbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc0b35a0 0x7f9cbc149040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9cbbfff640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc0b35a0 0x7f9cbc149040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58646/0 (socket says 172.21.15.16:58646) 2024-04-24T23:51:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9cbbfff640 1 -- 172.21.15.16:0/1578773631 learned_addr learned my addr 172.21.15.16:0/1578773631 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:51:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9cbb7fe640 1 --2- 172.21.15.16:0/1578773631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cbc148a40 0x7f9cbc0b1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9cbaffd640 1 --2- 172.21.15.16:0/1578773631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc0aec20 0x7f9cbc0b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9cbbfff640 1 -- 172.21.15.16:0/1578773631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc0aec20 msgr2=0x7f9cbc0b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9cbbfff640 1 --2- 172.21.15.16:0/1578773631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc0aec20 0x7f9cbc0b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9cbbfff640 1 -- 172.21.15.16:0/1578773631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cbc148a40 msgr2=0x7f9cbc0b1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9cbbfff640 1 --2- 172.21.15.16:0/1578773631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cbc148a40 0x7f9cbc0b1440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9cbbfff640 1 -- 172.21.15.16:0/1578773631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cbc0ad8e0 con 0x7f9cbc0b35a0 2024-04-24T23:51:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9cbbfff640 1 --2- 172.21.15.16:0/1578773631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc0b35a0 0x7f9cbc149040 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400e290 tx=0x7f9cb4009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:51:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9ca3fff640 1 -- 172.21.15.16:0/1578773631 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb4034030 con 0x7f9cbc0b35a0 2024-04-24T23:51:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9ca3fff640 1 -- 172.21.15.16:0/1578773631 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9cb4030e50 con 0x7f9cbc0b35a0 2024-04-24T23:51:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/1578773631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cbc0adb80 con 0x7f9cbc0b35a0 2024-04-24T23:51:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9ca3fff640 1 -- 172.21.15.16:0/1578773631 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cb4009c10 con 0x7f9cbc0b35a0 2024-04-24T23:51:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.874+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/1578773631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cbc0ae140 con 0x7f9cbc0b35a0 2024-04-24T23:51:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.875+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/1578773631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cbc068f90 con 0x7f9cbc0b35a0 2024-04-24T23:51:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.876+0000 7f9ca3fff640 1 -- 172.21.15.16:0/1578773631 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f9cb4037020 con 0x7f9cbc0b35a0 2024-04-24T23:51:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.876+0000 7f9ca3fff640 1 --2- 172.21.15.16:0/1578773631 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9ca806be30 0x7f9ca806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.876+0000 7f9ca3fff640 1 -- 172.21.15.16:0/1578773631 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f9cb40ab170 con 0x7f9cbc0b35a0 2024-04-24T23:51:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.877+0000 7f9cbb7fe640 1 --2- 172.21.15.16:0/1578773631 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9ca806be30 0x7f9ca806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.877+0000 7f9cbb7fe640 1 --2- 172.21.15.16:0/1578773631 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9ca806be30 0x7f9ca806e2f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9cac006fd0 tx=0x7f9cac008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:51:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:14.880+0000 7f9ca3fff640 1 -- 172.21.15.16:0/1578773631 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f9cb40795f0 con 0x7f9cbc0b35a0 2024-04-24T23:51:15.062 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.062+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/1578773631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f9cbc154f70 con 0x7f9cbc0b35a0 2024-04-24T23:51:15.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.065+0000 7f9ca3fff640 1 -- 172.21.15.16:0/1578773631 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v12)=0 v12) ==== 123+0+0 (secure 0 0 0) 0x7f9cb407d8f0 con 0x7f9cbc0b35a0 2024-04-24T23:51:15.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.066+0000 7f9ca3fff640 1 -- 172.21.15.16:0/1578773631 <== mon.0 v2:172.21.15.16:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f9cb4082120 con 0x7f9cbc0b35a0 2024-04-24T23:51:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.068+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/1578773631 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9ca806be30 msgr2=0x7f9ca806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.068+0000 7f9cc1b4f640 1 --2- 172.21.15.16:0/1578773631 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9ca806be30 0x7f9ca806e2f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f9cac006fd0 tx=0x7f9cac008040 comp rx=0 tx=0).stop 2024-04-24T23:51:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.068+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/1578773631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc0b35a0 msgr2=0x7f9cbc149040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.068+0000 7f9cc1b4f640 1 --2- 172.21.15.16:0/1578773631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc0b35a0 0x7f9cbc149040 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f9cb400e290 tx=0x7f9cb4009750 comp rx=0 tx=0).stop 2024-04-24T23:51:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.068+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/1578773631 shutdown_connections 2024-04-24T23:51:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.068+0000 7f9cc1b4f640 1 --2- 172.21.15.16:0/1578773631 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f9ca806be30 0x7f9ca806e2f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.068+0000 7f9cc1b4f640 1 --2- 172.21.15.16:0/1578773631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cbc0b35a0 0x7f9cbc149040 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.068+0000 7f9cc1b4f640 1 --2- 172.21.15.16:0/1578773631 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f9cbc0aec20 0x7f9cbc0b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.068+0000 7f9cc1b4f640 1 --2- 172.21.15.16:0/1578773631 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9cbc148a40 0x7f9cbc0b1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:15.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.068+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/1578773631 >> 172.21.15.16:0/1578773631 conn(0x7f9cbc06c200 msgr2=0x7f9cbc15ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:51:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.068+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/1578773631 shutdown_connections 2024-04-24T23:51:15.069 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.068+0000 7f9cc1b4f640 1 -- 172.21.15.16:0/1578773631 wait complete. 2024-04-24T23:51:15.078 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T23:51:15.079 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T23:51:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.180+0000 7f2871cee640 1 Processor -- start 2024-04-24T23:51:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.180+0000 7f2871cee640 1 -- start start 2024-04-24T23:51:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.181+0000 7f2871cee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f286c1457b0 0x7f286c0ce680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.181+0000 7f2871cee640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f286c15f990 0x7f286c0cebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.181+0000 7f2871cee640 1 -- --> v1:172.21.15.156:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286c15fe90 con 0x7f286c144b20 2024-04-24T23:51:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.181+0000 7f2871cee640 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 -- 0x7f286c06ea50 con 0x7f286c15f990 2024-04-24T23:51:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.181+0000 7f2871cee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f286c160170 con 0x7f286c1457b0 2024-04-24T23:51:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.181+0000 7f286bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f286c15f990 0x7f286c0cebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.181+0000 7f286bfff640 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f286c15f990 0x7f286c0cebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58648/0 (socket says 172.21.15.16:58648) 2024-04-24T23:51:15.182 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.181+0000 7f286bfff640 1 -- 172.21.15.16:0/1379207460 learned_addr learned my addr 172.21.15.16:0/1379207460 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:51:15.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.182+0000 7f286bfff640 1 -- 172.21.15.16:0/1379207460 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f286c1457b0 msgr2=0x7f286c0ce680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:51:15.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.182+0000 7f286affd640 1 --2- 172.21.15.16:0/1379207460 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f286c1457b0 0x7f286c0ce680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:15.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.182+0000 7f286bfff640 1 --2- 172.21.15.16:0/1379207460 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f286c1457b0 0x7f286c0ce680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:15.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.182+0000 7f286bfff640 1 -- 172.21.15.16:0/1379207460 >> v1:172.21.15.156:6789/0 conn(0x7f286c144b20 legacy=0x7f286c144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T23:51:15.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.182+0000 7f286bfff640 1 -- 172.21.15.16:0/1379207460 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286c160450 con 0x7f286c15f990 2024-04-24T23:51:15.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.182+0000 7f286affd640 1 --2- 172.21.15.16:0/1379207460 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f286c1457b0 0x7f286c0ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T23:51:15.183 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.182+0000 7f286bfff640 1 --2- 172.21.15.16:0/1379207460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f286c15f990 0x7f286c0cebc0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f2864009e50 tx=0x7f286402a630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c8ebeac4cd7cf7f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:51:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.183+0000 7f286a7fc640 1 -- 172.21.15.16:0/1379207460 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2864036070 con 0x7f286c15f990 2024-04-24T23:51:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.183+0000 7f286a7fc640 1 -- 172.21.15.16:0/1379207460 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f2864031600 con 0x7f286c15f990 2024-04-24T23:51:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.183+0000 7f286a7fc640 1 -- 172.21.15.16:0/1379207460 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28640319f0 con 0x7f286c15f990 2024-04-24T23:51:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.183+0000 7f2871cee640 1 -- 172.21.15.16:0/1379207460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f286c15f990 msgr2=0x7f286c0cebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.183+0000 7f2871cee640 1 --2- 172.21.15.16:0/1379207460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f286c15f990 0x7f286c0cebc0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f2864009e50 tx=0x7f286402a630 comp rx=0 tx=0).stop 2024-04-24T23:51:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.183+0000 7f2871cee640 1 -- 172.21.15.16:0/1379207460 shutdown_connections 2024-04-24T23:51:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.183+0000 7f2871cee640 1 --2- 172.21.15.16:0/1379207460 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f286c1457b0 0x7f286c0ce680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.183+0000 7f2871cee640 1 --2- 172.21.15.16:0/1379207460 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f286c15f990 0x7f286c0cebc0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.183+0000 7f2871cee640 1 -- 172.21.15.16:0/1379207460 >> 172.21.15.16:0/1379207460 conn(0x7f286c0b2000 msgr2=0x7f286c0cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:51:15.186 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.183+0000 7f2871cee640 1 -- 172.21.15.16:0/1379207460 shutdown_connections 2024-04-24T23:51:15.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.183+0000 7f2871cee640 1 -- 172.21.15.16:0/1379207460 wait complete. 2024-04-24T23:51:15.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.184+0000 7f2871cee640 1 Processor -- start 2024-04-24T23:51:15.187 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.185+0000 7f2871cee640 1 -- start start 2024-04-24T23:51:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.185+0000 7f2871cee640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f286c144b20 0x7f286c15c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.185+0000 7f2871cee640 1 --2- >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f286c1457b0 0x7f286c13dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.185+0000 7f2871cee640 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f286c15f990 0x7f286c156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.185+0000 7f2871cee640 1 -- --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 -- 0x7f286c0b3390 con 0x7f286c144b20 2024-04-24T23:51:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.185+0000 7f2871cee640 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f286c06d4b0 con 0x7f286c15f990 2024-04-24T23:51:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.185+0000 7f2871cee640 1 -- --> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] -- mon_getmap magic: 0 -- 0x7f286c06da40 con 0x7f286c1457b0 2024-04-24T23:51:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.185+0000 7f286b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f286c144b20 0x7f286c15c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.185+0000 7f286b7fe640 1 --2- >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f286c144b20 0x7f286c15c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58654/0 (socket says 172.21.15.16:58654) 2024-04-24T23:51:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.185+0000 7f286b7fe640 1 -- 172.21.15.16:0/878464245 learned_addr learned my addr 172.21.15.16:0/878464245 (peer_addr_for_me v2:172.21.15.16:0/0) 2024-04-24T23:51:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.186+0000 7f286bfff640 1 --2- 172.21.15.16:0/878464245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f286c15f990 0x7f286c156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.186+0000 7f286affd640 1 --2- 172.21.15.16:0/878464245 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f286c1457b0 0x7f286c13dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.186+0000 7f286b7fe640 1 -- 172.21.15.16:0/878464245 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f286c1457b0 msgr2=0x7f286c13dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:15.188 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.186+0000 7f286b7fe640 1 --2- 172.21.15.16:0/878464245 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f286c1457b0 0x7f286c13dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:15.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.186+0000 7f286b7fe640 1 -- 172.21.15.16:0/878464245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f286c15f990 msgr2=0x7f286c156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:15.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.186+0000 7f286b7fe640 1 --2- 172.21.15.16:0/878464245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f286c15f990 0x7f286c156ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:15.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.186+0000 7f286b7fe640 1 -- 172.21.15.16:0/878464245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286c13c470 con 0x7f286c144b20 2024-04-24T23:51:15.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.186+0000 7f286b7fe640 1 --2- 172.21.15.16:0/878464245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f286c144b20 0x7f286c15c490 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f285c00c430 tx=0x7f285c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:51:15.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.186+0000 7f2853fff640 1 -- 172.21.15.16:0/878464245 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c011790 con 0x7f286c144b20 2024-04-24T23:51:15.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.186+0000 7f2853fff640 1 -- 172.21.15.16:0/878464245 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f285c017070 con 0x7f286c144b20 2024-04-24T23:51:15.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.186+0000 7f2871cee640 1 -- 172.21.15.16:0/878464245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f286c13c750 con 0x7f286c144b20 2024-04-24T23:51:15.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.186+0000 7f2853fff640 1 -- 172.21.15.16:0/878464245 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c011bd0 con 0x7f286c144b20 2024-04-24T23:51:15.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.186+0000 7f2871cee640 1 -- 172.21.15.16:0/878464245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f286c15d3a0 con 0x7f286c144b20 2024-04-24T23:51:15.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.187+0000 7f2871cee640 1 -- 172.21.15.16:0/878464245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f286c0731f0 con 0x7f286c144b20 2024-04-24T23:51:15.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.188+0000 7f2853fff640 1 -- 172.21.15.16:0/878464245 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f285c01c460 con 0x7f286c144b20 2024-04-24T23:51:15.190 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.188+0000 7f2853fff640 1 --2- 172.21.15.16:0/878464245 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f285806be30 0x7f285806e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T23:51:15.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.188+0000 7f2853fff640 1 -- 172.21.15.16:0/878464245 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5421+0+0 (secure 0 0 0) 0x7f285c08aa10 con 0x7f286c144b20 2024-04-24T23:51:15.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.188+0000 7f286affd640 1 --2- 172.21.15.16:0/878464245 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f285806be30 0x7f285806e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T23:51:15.191 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.189+0000 7f286affd640 1 --2- 172.21.15.16:0/878464245 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f285806be30 0x7f285806e2f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f286c15bd80 tx=0x7f2860008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T23:51:15.192 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.192+0000 7f2853fff640 1 -- 172.21.15.16:0/878464245 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185023 (secure 0 0 0) 0x7f285c058f20 con 0x7f286c144b20 2024-04-24T23:51:15.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.417+0000 7f2871cee640 1 -- 172.21.15.16:0/878464245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f286c04f5e0 con 0x7f286c144b20 2024-04-24T23:51:15.417 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.418+0000 7f2853fff640 1 -- 172.21.15.16:0/878464245 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+3869 (secure 0 0 0) 0x7f285c05d220 con 0x7f286c144b20 2024-04-24T23:51:15.418 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-04-24T23:51:15.418 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":25,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5299,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6835/448535992","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6833","nonce":448535992},{"type":"v1","addr":"172.21.15.16:6835","nonce":448535992}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5305,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6836/1890687780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1890687780},{"type":"v1","addr":"172.21.15.156:6836","nonce":1890687780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5308,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2538147395","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6835","nonce":2538147395},{"type":"v1","addr":"172.21.15.156:6837","nonce":2538147395}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":5904,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6834/713211780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":713211780},{"type":"v1","addr":"172.21.15.16:6834","nonce":713211780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[]} 2024-04-24T23:51:15.418 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 25 2024-04-24T23:51:15.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.420+0000 7f2871cee640 1 -- 172.21.15.16:0/878464245 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f285806be30 msgr2=0x7f285806e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:15.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.420+0000 7f2871cee640 1 --2- 172.21.15.16:0/878464245 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f285806be30 0x7f285806e2f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f286c15bd80 tx=0x7f2860008040 comp rx=0 tx=0).stop 2024-04-24T23:51:15.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.420+0000 7f2871cee640 1 -- 172.21.15.16:0/878464245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f286c144b20 msgr2=0x7f286c15c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T23:51:15.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.420+0000 7f2871cee640 1 --2- 172.21.15.16:0/878464245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f286c144b20 0x7f286c15c490 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f285c00c430 tx=0x7f285c00c8f0 comp rx=0 tx=0).stop 2024-04-24T23:51:15.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.420+0000 7f2871cee640 1 -- 172.21.15.16:0/878464245 shutdown_connections 2024-04-24T23:51:15.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.420+0000 7f2871cee640 1 --2- 172.21.15.16:0/878464245 >> [v2:172.21.15.156:6832/3478836820,v1:172.21.15.156:6833/3478836820] conn(0x7f285806be30 0x7f285806e2f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:15.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.420+0000 7f2871cee640 1 --2- 172.21.15.16:0/878464245 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f286c15f990 0x7f286c156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:15.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.420+0000 7f2871cee640 1 --2- 172.21.15.16:0/878464245 >> [v2:172.21.15.156:3301/0,v1:172.21.15.156:6790/0] conn(0x7f286c1457b0 0x7f286c13dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:15.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.420+0000 7f2871cee640 1 --2- 172.21.15.16:0/878464245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f286c144b20 0x7f286c15c490 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T23:51:15.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.420+0000 7f2871cee640 1 -- 172.21.15.16:0/878464245 >> 172.21.15.16:0/878464245 conn(0x7f286c0b2000 msgr2=0x7f286c0b3510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T23:51:15.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.420+0000 7f2871cee640 1 -- 172.21.15.16:0/878464245 shutdown_connections 2024-04-24T23:51:15.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-04-24T23:51:15.420+0000 7f2871cee640 1 -- 172.21.15.16:0/878464245 wait complete. 2024-04-24T23:51:15.431 DEBUG:tasks.cephfs.filesystem:already dead... 2024-04-24T23:51:15.431 INFO:teuthology.misc:Shutting down mds daemons... 2024-04-24T23:51:15.431 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-04-24T23:51:15.431 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:15.440 INFO:tasks.ceph.mds.a:Stopped 2024-04-24T23:51:15.441 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-04-24T23:51:15.441 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:15.490 INFO:tasks.ceph.mds.c:Stopped 2024-04-24T23:51:15.490 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-04-24T23:51:15.490 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:15.502 INFO:tasks.ceph.mds.b:Stopped 2024-04-24T23:51:15.502 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-04-24T23:51:15.502 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:15.554 INFO:tasks.ceph.mds.d:Stopped 2024-04-24T23:51:15.554 INFO:teuthology.misc:Shutting down osd daemons... 2024-04-24T23:51:15.554 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-04-24T23:51:15.554 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:15.584 INFO:tasks.ceph.osd.0:Stopped 2024-04-24T23:51:15.584 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-04-24T23:51:15.584 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:15.659 INFO:tasks.ceph.osd.1:Stopped 2024-04-24T23:51:15.659 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-04-24T23:51:15.659 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:15.726 INFO:tasks.ceph.osd.2:Stopped 2024-04-24T23:51:15.727 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-04-24T23:51:15.727 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:15.793 INFO:tasks.ceph.osd.3:Stopped 2024-04-24T23:51:15.794 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-04-24T23:51:15.794 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:15.821 INFO:tasks.ceph.osd.4:Stopped 2024-04-24T23:51:15.822 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-04-24T23:51:15.822 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:15.895 INFO:tasks.ceph.osd.5:Stopped 2024-04-24T23:51:15.895 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-04-24T23:51:15.896 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:15.962 INFO:tasks.ceph.osd.6:Stopped 2024-04-24T23:51:15.962 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-04-24T23:51:15.962 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:16.029 INFO:tasks.ceph.osd.7:Stopped 2024-04-24T23:51:16.029 INFO:teuthology.misc:Shutting down mgr daemons... 2024-04-24T23:51:16.030 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-04-24T23:51:16.030 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:16.055 INFO:tasks.ceph.mgr.y:Stopped 2024-04-24T23:51:16.056 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-04-24T23:51:16.056 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:16.098 INFO:tasks.ceph.mgr.x:Stopped 2024-04-24T23:51:16.099 INFO:teuthology.misc:Shutting down mon daemons... 2024-04-24T23:51:16.099 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-04-24T23:51:16.099 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:16.111 INFO:tasks.ceph.mon.a:Stopped 2024-04-24T23:51:16.111 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-04-24T23:51:16.111 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:16.150 INFO:tasks.ceph.mon.b:Stopped 2024-04-24T23:51:16.151 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-04-24T23:51:16.151 INFO:teuthology.orchestra.run:waiting for 300 2024-04-24T23:51:16.203 INFO:tasks.ceph.mon.c:Stopped 2024-04-24T23:51:16.203 INFO:tasks.ceph:Checking cluster log for badness... 2024-04-24T23:51:16.203 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-04-24T23:51:16.236 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:51:16.236 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-04-24T23:51:16.423 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:51:16.423 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-04-24T23:51:16.522 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:51:16.522 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-04-24T23:51:16.596 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:51:16.596 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-04-24T23:51:16.646 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:51:16.646 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-04-24T23:51:16.823 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:51:16.823 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-04-24T23:51:16.887 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:51:16.887 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-04-24T23:51:16.938 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:51:16.938 DEBUG:teuthology.orchestra.run.smithi156:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-04-24T23:51:16.994 INFO:tasks.ceph:Archiving mon data... 2024-04-24T23:51:16.996 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219/data/mon.a.tgz 2024-04-24T23:51:16.996 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-04-24T23:51:17.012 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.0TyymECsoA 2024-04-24T23:51:17.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0TyymECsoA 2024-04-24T23:51:17.166 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.0TyymECsoA 2024-04-24T23:51:17.249 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.0TyymECsoA is 317KB 2024-04-24T23:51:17.318 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.0TyymECsoA 2024-04-24T23:51:17.334 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219/data/mon.b.tgz 2024-04-24T23:51:17.334 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2024-04-24T23:51:17.351 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.fJYsUcepaP 2024-04-24T23:51:17.352 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.fJYsUcepaP 2024-04-24T23:51:17.533 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.fJYsUcepaP 2024-04-24T23:51:17.615 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.fJYsUcepaP is 420KB 2024-04-24T23:51:17.714 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.fJYsUcepaP 2024-04-24T23:51:17.729 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219/data/mon.c.tgz 2024-04-24T23:51:17.729 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2024-04-24T23:51:17.782 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.y8dhWGeIkq 2024-04-24T23:51:17.783 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.y8dhWGeIkq 2024-04-24T23:51:17.923 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.y8dhWGeIkq 2024-04-24T23:51:18.006 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.y8dhWGeIkq is 420KB 2024-04-24T23:51:18.103 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.y8dhWGeIkq 2024-04-24T23:51:18.117 INFO:tasks.ceph:Cleaning ceph cluster... 2024-04-24T23:51:18.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T23:51:18.120 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-04-24T23:51:18.201 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-24T23:51:18.236 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-04-24T23:51:18.236 INFO:tasks.ceph:Archiving crash dumps... 2024-04-24T23:51:18.238 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219/remote/smithi016/crash 2024-04-24T23:51:18.239 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T23:51:18.268 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219/remote/smithi156/crash 2024-04-24T23:51:18.269 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-04-24T23:51:18.299 INFO:tasks.ceph:Shutting down logrotate 2024-04-24T23:51:18.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:51:18.313 DEBUG:teuthology.orchestra.run.smithi156:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-04-24T23:51:18.393 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-24T23:51:18.419 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-04-24T23:51:18.442 INFO:tasks.ceph:Compressing logs... 2024-04-24T23:51:18.443 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T23:51:18.462 DEBUG:teuthology.orchestra.run.smithi156:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T23:51:18.537 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-24T23:51:18.537 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34701.log 2024-04-24T23:51:18.537 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.40732.log 2024-04-24T23:51:18.537 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.34701.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-24T23:51:18.537 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.1.40732.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.34701.log.gz 2024-04-24T23:51:18.537 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34741.log 2024-04-24T23:51:18.537 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-04-24T23:51:18.537 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph.audit.log: 91.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-04-24T23:51:18.537 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.34741.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-04-24T23:51:18.538 INFO:teuthology.orchestra.run.smithi156.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.1.40732.log.gz 2024-04-24T23:51:18.538 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.7.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34741.log.gz 2024-04-24T23:51:18.538 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-04-24T23:51:18.538 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-04-24T23:51:18.538 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34621.log 2024-04-24T23:51:18.538 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34450.log 2024-04-24T23:51:18.538 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.34621.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-04-24T23:51:18.538 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.34450.log: gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34621.log.gz 2024-04-24T23:51:18.538 INFO:teuthology.orchestra.run.smithi156.stderr: --verbose -- /var/log/ceph/ceph-mon.c.log 2024-04-24T23:51:18.538 INFO:teuthology.orchestra.run.smithi156.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-24T23:51:18.538 INFO:teuthology.orchestra.run.smithi156.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34450.log.gz 2024-04-24T23:51:18.538 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5/var/log/ceph/ceph-mgr.x.log: --verbose -- /var/log/ceph/ceph-osd.6.log 2024-04-24T23:51:18.539 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34530.log 2024-04-24T23:51:18.539 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.41189.log 2024-04-24T23:51:18.539 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.34530.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.34530.log.gz 2024-04-24T23:51:18.539 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34490.log 2024-04-24T23:51:18.539 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.3.41189.log: 91.2% -- replaced with /var/log/ceph/ceph-client.3.41189.log.gz 2024-04-24T23:51:18.539 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34574.log 2024-04-24T23:51:18.539 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.34490.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34490.log.gz 2024-04-24T23:51:18.539 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34661.log 2024-04-24T23:51:18.539 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.34574.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34574.log.gz 2024-04-24T23:51:18.539 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-04-24T23:51:18.540 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-client.admin.34661.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.34661.log.gz 2024-04-24T23:51:18.540 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-04-24T23:51:18.540 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mds.d.log: 90.9% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-04-24T23:51:18.543 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/ceph-mon.b.log: 92.0% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-04-24T23:51:18.564 INFO:teuthology.orchestra.run.smithi156.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-04-24T23:51:18.608 INFO:teuthology.orchestra.run.smithi156.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-04-24T23:51:18.645 INFO:teuthology.orchestra.run.smithi156.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-04-24T23:51:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46064.log 2024-04-24T23:51:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43194.log 2024-04-24T23:51:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50315.log 2024-04-24T23:51:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46064.log: /var/log/ceph/ceph-client.admin.43194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49625.log 2024-04-24T23:51:18.652 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46064.log.gz 2024-04-24T23:51:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50315.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43194.log.gz 2024-04-24T23:51:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40450.log 2024-04-24T23:51:18.652 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50315.log.gz 2024-04-24T23:51:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42303.log 2024-04-24T23:51:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45602.log 2024-04-24T23:51:18.652 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49625.log.gz 2024-04-24T23:51:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48041.log 2024-04-24T23:51:18.652 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40450.log.gz 2024-04-24T23:51:18.652 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42303.log.gz 2024-04-24T23:51:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46316.log 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46442.log 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45602.log.gz 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4%/var/log/ceph/ceph-client.admin.46316.log: -- replaced with /var/log/ceph/ceph-client.admin.48041.log.gz 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42051.log 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49709.log 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46316.log.gz 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42051.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46442.log.gz 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41925.log 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42051.log.gz 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47429.log 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49709.log: /var/log/ceph/ceph-client.admin.41925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50134.log 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49709.log.gz 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41925.log.gz/var/log/ceph/ceph-client.admin.47429.log: 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48209.log 2024-04-24T23:51:18.653 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.47429.log.gz 2024-04-24T23:51:18.654 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53150.log 2024-04-24T23:51:18.654 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46747.log 2024-04-24T23:51:18.654 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50134.log.gz 2024-04-24T23:51:18.654 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49422.log 2024-04-24T23:51:18.654 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48209.log.gz 2024-04-24T23:51:18.654 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53150.log.gz 2024-04-24T23:51:18.654 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48083.log 2024-04-24T23:51:18.654 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46747.log.gz 2024-04-24T23:51:18.654 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42521.log 2024-04-24T23:51:18.654 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48083.log: gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49422.log.gz -5 2024-04-24T23:51:18.654 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45896.log 2024-04-24T23:51:18.654 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48083.log.gz 2024-04-24T23:51:18.654 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49296.log 2024-04-24T23:51:18.654 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45896.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42521.log.gz 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40534.log 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45896.log.gz 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42524.log 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40534.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.49296.log.gz 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41369.log 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42524.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40534.log.gzgzip 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43467.log 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41369.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42524.log.gz 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50269.log 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43467.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41369.log.gz 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46873.log 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43467.log.gz 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34794.log 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50269.log.gz 2024-04-24T23:51:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46148.log 2024-04-24T23:51:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34794.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46873.log.gz 2024-04-24T23:51:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48582.log 2024-04-24T23:51:18.656 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34794.log.gz 2024-04-24T23:51:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52844.log 2024-04-24T23:51:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48582.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46148.log.gz 2024-04-24T23:51:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52521.log 2024-04-24T23:51:18.656 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48582.log.gz 2024-04-24T23:51:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46526.log 2024-04-24T23:51:18.656 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52844.log.gz 2024-04-24T23:51:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52521.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-04-24T23:51:18.656 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52521.log.gz 2024-04-24T23:51:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52335.log 2024-04-24T23:51:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46526.log.gz 2024-04-24T23:51:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49667.log 2024-04-24T23:51:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43198.log 2024-04-24T23:51:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49667.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52335.log.gzgzip 2024-04-24T23:51:18.657 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- 84.8% /var/log/ceph/ceph-client.admin.50795.log 2024-04-24T23:51:18.657 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-04-24T23:51:18.657 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49667.log.gz 2024-04-24T23:51:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50357.log 2024-04-24T23:51:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50795.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43198.log.gz 2024-04-24T23:51:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49794.log 2024-04-24T23:51:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46568.log 2024-04-24T23:51:18.657 INFO:teuthology.orchestra.run.smithi016.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.50795.log.gz 2024-04-24T23:51:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49794.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-04-24T23:51:18.658 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50357.log.gz 2024-04-24T23:51:18.658 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49794.log.gz 2024-04-24T23:51:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49086.log 2024-04-24T23:51:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.2.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46568.log.gz 2024-04-24T23:51:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42177.log 2024-04-24T23:51:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46022.log 2024-04-24T23:51:18.658 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49086.log.gz 2024-04-24T23:51:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51511.log 2024-04-24T23:51:18.658 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42177.log.gz 2024-04-24T23:51:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41453.log 2024-04-24T23:51:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51511.log: gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.46022.log.gz 2024-04-24T23:51:18.658 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42261.log 2024-04-24T23:51:18.658 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51511.log.gz 2024-04-24T23:51:18.658 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42856.log 2024-04-24T23:51:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose 88.9% -- /var/log/ceph/ceph-client.admin.46190.log -- replaced with /var/log/ceph/ceph-client.admin.41453.log.gz 2024-04-24T23:51:18.659 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-24T23:51:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42261.log: /var/log/ceph/ceph-client.admin.42856.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42261.log.gz 2024-04-24T23:51:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42844.log 2024-04-24T23:51:18.659 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42856.log.gz 2024-04-24T23:51:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40768.log 2024-04-24T23:51:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42844.log: gzip -5 --verbose 89.0% -- /var/log/ceph/ceph-client.admin.42841.log -- replaced with /var/log/ceph/ceph-client.admin.46190.log.gz 2024-04-24T23:51:18.659 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-24T23:51:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40768.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42844.log.gz 2024-04-24T23:51:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51372.log 2024-04-24T23:51:18.659 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40768.log.gz 2024-04-24T23:51:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40408.log 2024-04-24T23:51:18.659 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42841.log.gz 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.51372.log: -- /var/log/ceph/ceph-client.admin.52167.log 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42527.log 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.51372.log.gz 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40408.log.gz 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51600.log 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52167.log.gz 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42527.log: /var/log/ceph/ceph-client.admin.51600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50004.log 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42527.log.gz 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.51600.log.gz 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45728.log 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51762.log 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45728.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50004.log.gz 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50525.log 2024-04-24T23:51:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40006.log 2024-04-24T23:51:18.661 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45728.log.gz 2024-04-24T23:51:18.661 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50525.log: 88.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-04-24T23:51:18.661 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51762.log.gz 2024-04-24T23:51:18.661 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48708.log 2024-04-24T23:51:18.661 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50525.log.gz 2024-04-24T23:51:18.661 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34703.log 2024-04-24T23:51:18.661 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40006.log: /var/log/ceph/ceph-client.admin.48708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39963.log 2024-04-24T23:51:18.661 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40006.log.gz 2024-04-24T23:51:18.661 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.48708.log.gz 2024-04-24T23:51:18.661 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42518.log 2024-04-24T23:51:18.661 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.34703.log.gz 2024-04-24T23:51:18.661 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53318.log 2024-04-24T23:51:18.662 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42518.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39963.log.gz 2024-04-24T23:51:18.662 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39663.log 2024-04-24T23:51:18.662 INFO:teuthology.orchestra.run.smithi016.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42518.log.gz 2024-04-24T23:51:18.662 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41967.log 2024-04-24T23:51:18.662 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53318.log.gz 2024-04-24T23:51:18.662 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.39663.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.52041.log 2024-04-24T23:51:18.662 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44822.log 2024-04-24T23:51:18.662 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39663.log.gz 2024-04-24T23:51:18.662 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52041.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41967.log.gz 2024-04-24T23:51:18.662 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50711.log 2024-04-24T23:51:18.662 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52041.log.gz 2024-04-24T23:51:18.662 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40618.log 2024-04-24T23:51:18.662 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44822.log: /var/log/ceph/ceph-client.admin.50711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43424.log 2024-04-24T23:51:18.662 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44822.log.gz 2024-04-24T23:51:18.663 INFO:teuthology.orchestra.run.smithi016.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.50711.log.gz 2024-04-24T23:51:18.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43188.log 2024-04-24T23:51:18.663 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40618.log.gz 2024-04-24T23:51:18.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49962.log 2024-04-24T23:51:18.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46274.log 2024-04-24T23:51:18.663 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43424.log.gz 2024-04-24T23:51:18.663 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43188.log.gz 2024-04-24T23:51:18.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53838.log 2024-04-24T23:51:18.663 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49962.log.gz 2024-04-24T23:51:18.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46274.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-04-24T23:51:18.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40090.log 2024-04-24T23:51:18.663 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46274.log.gz 2024-04-24T23:51:18.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.3.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53838.log.gz 2024-04-24T23:51:18.663 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45560.log 2024-04-24T23:51:18.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45518.log 2024-04-24T23:51:18.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45560.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40090.log.gz 2024-04-24T23:51:18.664 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43719.log 2024-04-24T23:51:18.664 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45560.log.gz 2024-04-24T23:51:18.664 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47899.log 2024-04-24T23:51:18.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45518.log: /var/log/ceph/ceph-client.admin.43719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41637.log 2024-04-24T23:51:18.664 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45518.log.gz 2024-04-24T23:51:18.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47899.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43719.log.gz 2024-04-24T23:51:18.664 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42525.log 2024-04-24T23:51:18.664 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47899.log.gz 2024-04-24T23:51:18.664 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41637.log: /var/log/ceph/ceph-client.admin.53192.log 2024-04-24T23:51:18.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42857.log 2024-04-24T23:51:18.664 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41637.log.gz 2024-04-24T23:51:18.664 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42525.log.gz 2024-04-24T23:51:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41763.log 2024-04-24T23:51:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52083.log 2024-04-24T23:51:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42857.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53192.log.gz 2024-04-24T23:51:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44309.log 2024-04-24T23:51:18.665 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42857.log.gz 2024-04-24T23:51:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52083.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41763.log.gz 2024-04-24T23:51:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43803.log 2024-04-24T23:51:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40660.log 2024-04-24T23:51:18.665 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52083.log.gz 2024-04-24T23:51:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43803.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44309.log.gz 2024-04-24T23:51:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50609.log 2024-04-24T23:51:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40660.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43803.log.gz 2024-04-24T23:51:18.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45980.log 2024-04-24T23:51:18.665 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40660.log.gz 2024-04-24T23:51:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40132.log 2024-04-24T23:51:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45980.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50609.log.gz 2024-04-24T23:51:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48750.log 2024-04-24T23:51:18.666 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45980.log.gz 2024-04-24T23:51:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43677.log 2024-04-24T23:51:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48750.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.40132.log.gz 2024-04-24T23:51:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46931.log 2024-04-24T23:51:18.666 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48750.log.gz 2024-04-24T23:51:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43677.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-04-24T23:51:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46931.log: gzip -5 --verbose 87.8% -- /var/log/ceph/ceph-client.admin.45686.log -- replaced with /var/log/ceph/ceph-client.admin.43677.log.gz 2024-04-24T23:51:18.666 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-24T23:51:18.666 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46931.log.gz 2024-04-24T23:51:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47941.log 2024-04-24T23:51:18.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43059.log 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2%/var/log/ceph/ceph-client.admin.47941.log: -- replaced with /var/log/ceph/ceph-client.admin.45686.log.gz 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43887.log 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47941.log.gz 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47857.log 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.43059.log.gz --verbose 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr: -- /var/log/ceph/ceph-client.admin.51930.log 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43887.log: /var/log/ceph/ceph-client.admin.47857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50185.log 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43887.log.gz 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51930.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47857.log.gz 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40048.log 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50185.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51930.log.gz 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49752.log 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50185.log.gz 2024-04-24T23:51:18.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40048.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-04-24T23:51:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51415.log 2024-04-24T23:51:18.668 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40048.log.gz 2024-04-24T23:51:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49752.log: /var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-04-24T23:51:18.668 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49752.log.gz 2024-04-24T23:51:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52648.log 2024-04-24T23:51:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51415.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45770.log 2024-04-24T23:51:18.668 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51415.log.gz 2024-04-24T23:51:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45476.log 2024-04-24T23:51:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52648.log: /var/log/ceph/ceph-client.admin.45770.log: gzip -5 --verbose 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52648.log.gz 2024-04-24T23:51:18.668 INFO:teuthology.orchestra.run.smithi016.stderr: -- /var/log/ceph/ceph-client.admin.48167.log 2024-04-24T23:51:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45476.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.45770.log.gz 2024-04-24T23:51:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52419.log 2024-04-24T23:51:18.668 INFO:teuthology.orchestra.run.smithi016.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45476.log.gz 2024-04-24T23:51:18.668 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40492.log 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52419.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48167.log.gz 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52377.log 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52419.log.gz 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40492.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52377.log: gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40492.log.gz 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.47983.log 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.0.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52377.log.gz 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47303.log 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44055.log 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47983.log.gz 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47303.log: 92.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46663.log 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47303.log.gz 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50753.log 2024-04-24T23:51:18.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46663.log: gzip -5 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44055.log.gz 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53796.log 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4%/var/log/ceph/ceph-client.admin.50753.log: -- replaced with /var/log/ceph/ceph-client.admin.46663.log.gz 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49002.log 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50753.log.gz 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44181.log 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49002.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.53796.log.gz 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40702.log 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49002.log.gz 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53444.log 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44181.log: /var/log/ceph/ceph-client.admin.40702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45323.log 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40218.log 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.44181.log.gz 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45323.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53444.log.gz 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.40702.log.gz 2024-04-24T23:51:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53108.log 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45323.log.gz 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46232.log 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52690.log 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40218.log.gz 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41178.log 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53108.log.gz/var/log/ceph/ceph-client.admin.46232.log: 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52690.log: gzip -5 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46232.log.gz 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48624.log 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41178.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52690.log.gz 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52886.log 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48624.log: gzip -5 --verbose 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48624.log.gz 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr: -- /var/log/ceph/ceph-client.admin.49212.log 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48251.log 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52886.log.gz 2024-04-24T23:51:18.671 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49254.log 2024-04-24T23:51:18.672 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49212.log.gz 2024-04-24T23:51:18.672 INFO:teuthology.orchestra.run.smithi016.stderr: 92.7%/var/log/ceph/ceph-client.admin.48251.log: -- replaced with /var/log/ceph/ceph-client.admin.41178.log.gz 2024-04-24T23:51:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53402.log 2024-04-24T23:51:18.672 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48251.log.gz 2024-04-24T23:51:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41579.log 2024-04-24T23:51:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.53402.log: 88.8% -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-04-24T23:51:18.672 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49254.log.gz 2024-04-24T23:51:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41579.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49044.log 2024-04-24T23:51:18.672 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53402.log.gz 2024-04-24T23:51:18.672 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41579.log.gz 2024-04-24T23:51:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41495.log 2024-04-24T23:51:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.49044.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.49380.log 2024-04-24T23:51:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41495.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49044.log.gz 2024-04-24T23:51:18.672 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53486.log 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41495.log.gz 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42517.log 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49380.log: /var/log/ceph/ceph-client.admin.53486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48540.log 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42517.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53486.log.gz 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49380.log.gz 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42517.log.gz 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50567.log 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.48540.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43593.log 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50567.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48540.log.gz 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46705.log 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50567.log.gz 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52606.log 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43593.log: /var/log/ceph/ceph-client.admin.46705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51553.log 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52606.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43593.log.gz 2024-04-24T23:51:18.673 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46705.log.gz 2024-04-24T23:51:18.674 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52606.log.gz 2024-04-24T23:51:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47773.log 2024-04-24T23:51:18.674 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51553.log.gz 2024-04-24T23:51:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51846.log 2024-04-24T23:51:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45366.log 2024-04-24T23:51:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51846.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47773.log.gz 2024-04-24T23:51:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48792.log 2024-04-24T23:51:18.674 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51846.log.gz 2024-04-24T23:51:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52461.log 2024-04-24T23:51:18.674 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45366.log.gz 2024-04-24T23:51:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44097.log 2024-04-24T23:51:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48792.log: /var/log/ceph/ceph-client.admin.52461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49338.log 2024-04-24T23:51:18.674 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44097.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48792.log.gz 2024-04-24T23:51:18.674 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52461.log.gz 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48960.log 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44097.log.gz 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40364.log 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49338.log: /var/log/ceph/ceph-client.admin.48960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40322.log 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40364.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48960.log.gz 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49338.log.gz 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40364.log.gz 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43197.log 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48125.log 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43197.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40322.log.gz 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45854.log 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43197.log.gz 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39921.log 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48125.log: /var/log/ceph/ceph-client.admin.45854.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48125.log.gz 2024-04-24T23:51:18.675 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45854.log.gz 2024-04-24T23:51:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42135.log 2024-04-24T23:51:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.51121.log 2024-04-24T23:51:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42135.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39921.log.gz 2024-04-24T23:51:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53756.log 2024-04-24T23:51:18.676 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42135.log.gz 2024-04-24T23:51:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.51121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51642.log 2024-04-24T23:51:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41847.log 2024-04-24T23:51:18.676 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53756.log.gz 2024-04-24T23:51:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51642.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.51642.log.gz 2024-04-24T23:51:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47099.log 2024-04-24T23:51:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44013.log 2024-04-24T23:51:18.676 INFO:teuthology.orchestra.run.smithi016.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.0.51121.log.gz 2024-04-24T23:51:18.676 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41847.log.gz 2024-04-24T23:51:18.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42858.log 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47099.log.gz 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46358.log 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44013.log.gz 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47615.log 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42858.log.gz 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48372.log 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46358.log: /var/log/ceph/ceph-client.admin.47615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42522.log 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48372.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46358.log.gz 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47615.log.gz 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47657.log 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48372.log.gz 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.42522.log: -- /var/log/ceph/ceph-client.admin.39840.log 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47657.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42522.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50483.log 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-24T23:51:18.677 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47657.log.gz 2024-04-24T23:51:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43635.log 2024-04-24T23:51:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48414.log 2024-04-24T23:51:18.678 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39840.log.gz 2024-04-24T23:51:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45812.log 89.4% 2024-04-24T23:51:18.678 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50483.log.gz 2024-04-24T23:51:18.678 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43635.log.gz 2024-04-24T23:51:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41721.log 2024-04-24T23:51:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48414.log: /var/log/ceph/ceph-client.admin.45812.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48414.log.gz 2024-04-24T23:51:18.678 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45812.log.gz 2024-04-24T23:51:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51457.log 2024-04-24T23:51:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53570.log 2024-04-24T23:51:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51457.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41721.log.gz 2024-04-24T23:51:18.678 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40280.log 2024-04-24T23:51:18.679 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51457.log.gz 2024-04-24T23:51:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48876.log 2024-04-24T23:51:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53570.log: /var/log/ceph/ceph-client.admin.40280.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53570.log.gz 2024-04-24T23:51:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40280.log.gz 2024-04-24T23:51:18.679 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46789.log 2024-04-24T23:51:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48876.log: gzip -5 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48876.log.gz 2024-04-24T23:51:18.679 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41805.log 2024-04-24T23:51:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42860.log 2024-04-24T23:51:18.679 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9%/var/log/ceph/ceph-client.admin.41805.log: -- replaced with /var/log/ceph/ceph-client.admin.46789.log.gz 2024-04-24T23:51:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50090.log 2024-04-24T23:51:18.679 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41805.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53612.log 2024-04-24T23:51:18.679 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-24T23:51:18.679 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42860.log: /var/log/ceph/ceph-client.admin.50090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46400.log 2024-04-24T23:51:18.679 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8%/var/log/ceph/ceph-client.admin.53612.log: -- replaced with /var/log/ceph/ceph-client.admin.42860.log.gz 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50090.log.gz 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53612.log.gz 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46973.log 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43551.log 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46973.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46400.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52125.log 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46973.log.gz 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50227.log 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44139.log 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.43551.log.gz 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7%/var/log/ceph/ceph-client.admin.50227.log: -- replaced with /var/log/ceph/ceph-client.admin.52125.log.gz 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43509.log 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7%/var/log/ceph/ceph-client.admin.44139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47057.log 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50227.log.gz 2024-04-24T23:51:18.680 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44139.log.gz 2024-04-24T23:51:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42345.log 2024-04-24T23:51:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47141.log 2024-04-24T23:51:18.681 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43509.log.gz 2024-04-24T23:51:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42345.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47057.log.gz 2024-04-24T23:51:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49170.log 2024-04-24T23:51:18.681 INFO:teuthology.orchestra.run.smithi016.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42345.log.gz 2024-04-24T23:51:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47345.log 2024-04-24T23:51:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47141.log: /var/log/ceph/ceph-client.admin.49170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42863.log 2024-04-24T23:51:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47345.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.47141.log.gz 2024-04-24T23:51:18.681 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49170.log.gz 2024-04-24T23:51:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47345.log.gz 2024-04-24T23:51:18.681 INFO:teuthology.orchestra.run.smithi016.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40174.log 2024-04-24T23:51:18.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51888.log 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42863.log.gz 2024-04-24T23:51:18.681 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-24T23:51:18.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41679.log 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40174.log.gz 2024-04-24T23:51:18.682 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-24T23:51:18.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51888.log: gzip -5 --verbose 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51888.log.gz 2024-04-24T23:51:18.682 INFO:teuthology.orchestra.run.smithi016.stderr: -- /var/log/ceph/ceph.tmp-client.admin.28410.log 2024-04-24T23:51:18.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50861.log 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41679.log.gz 2024-04-24T23:51:18.682 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-24T23:51:18.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.tmp-client.admin.28410.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28410.log.gz 2024-04-24T23:51:18.682 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48330.log 2024-04-24T23:51:18.682 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.50861.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47489.log 2024-04-24T23:51:18.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50441.log 2024-04-24T23:51:18.682 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50861.log.gz 2024-04-24T23:51:18.682 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6%/var/log/ceph/ceph-client.admin.47489.log: -- replaced with /var/log/ceph/ceph-client.admin.48330.log.gz 2024-04-24T23:51:18.682 INFO:teuthology.orchestra.run.smithi016.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47489.log.gz 2024-04-24T23:51:18.682 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43845.log 2024-04-24T23:51:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49836.log 2024-04-24T23:51:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43845.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50441.log.gz 2024-04-24T23:51:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53234.log 2024-04-24T23:51:18.683 INFO:teuthology.orchestra.run.smithi016.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43845.log.gz 2024-04-24T23:51:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53654.log 2024-04-24T23:51:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49836.log: /var/log/ceph/ceph-client.admin.53234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43761.log 2024-04-24T23:51:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53654.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49836.log.gz 2024-04-24T23:51:18.683 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53234.log.gz 2024-04-24T23:51:18.683 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53360.log 2024-04-24T23:51:18.683 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53654.log.gz 2024-04-24T23:51:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.45199.log 2024-04-24T23:51:18.683 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49878.log 2024-04-24T23:51:18.683 INFO:teuthology.orchestra.run.smithi016.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43761.log.gz 2024-04-24T23:51:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53360.log: /var/log/ceph/ceph-client.2.45199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47815.log 2024-04-24T23:51:18.684 INFO:teuthology.orchestra.run.smithi016.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53360.log.gz 2024-04-24T23:51:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48498.log 2024-04-24T23:51:18.684 INFO:teuthology.orchestra.run.smithi016.stderr: 91.1% 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49878.log.gz 2024-04-24T23:51:18.684 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.2.45199.log.gz 2024-04-24T23:51:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48918.log 2024-04-24T23:51:18.684 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47815.log.gz/var/log/ceph/ceph-client.admin.48498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48666.log 2024-04-24T23:51:18.684 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-24T23:51:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48918.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48498.log.gz 2024-04-24T23:51:18.684 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42219.log 2024-04-24T23:51:18.684 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48918.log.gz 2024-04-24T23:51:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47261.log 2024-04-24T23:51:18.684 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48834.log 2024-04-24T23:51:18.684 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48666.log.gz 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42219.log: /var/log/ceph/ceph-client.admin.47261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53276.log 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1%/var/log/ceph/ceph-client.admin.48834.log: -- replaced with /var/log/ceph/ceph-client.admin.42219.log.gz 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47261.log.gz 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48834.log.gz 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51804.log 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46484.log 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51804.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53276.log.gz 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42861.log 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46484.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51804.log.gz 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49920.log 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46484.log.gz 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53024.log 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.42861.log.gz 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39744.log 2024-04-24T23:51:18.685 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49920.log: /var/log/ceph/ceph-client.admin.53024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52971.log 2024-04-24T23:51:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39744.log: 88.3% 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49920.log.gz 2024-04-24T23:51:18.686 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53024.log.gz 2024-04-24T23:51:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.39744.log.gz --verbose 2024-04-24T23:51:18.686 INFO:teuthology.orchestra.run.smithi016.stderr: -- /var/log/ceph/ceph-client.admin.47573.log 2024-04-24T23:51:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:gzip/var/log/ceph/ceph-client.admin.52971.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47387.log 2024-04-24T23:51:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47573.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52971.log.gz 2024-04-24T23:51:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-04-24T23:51:18.686 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47573.log.gz 2024-04-24T23:51:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.47387.log: -- /var/log/ceph/ceph-client.admin.52209.log 2024-04-24T23:51:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mds.a.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46106.log 2024-04-24T23:51:18.686 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47387.log.gz 2024-04-24T23:51:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52293.log 2024-04-24T23:51:18.686 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52209.log: /var/log/ceph/ceph-client.admin.46106.log: 89.5% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46106.log.gz 2024-04-24T23:51:18.686 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52209.log.gz 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43190.log 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41411.log 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52293.log.gz 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43971.log 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41411.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43190.log.gz 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41411.log.gz 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34427.log 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48456.log 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43971.log.gz/var/log/ceph/ceph-client.admin.34427.log: 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.44700.log 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.34427.log.gz/var/log/ceph/ceph-client.admin.48456.log: 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48456.log.gz 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43199.log 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.0.44700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51312.log 2024-04-24T23:51:18.687 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43199.log: 90.2% -- replaced with /var/log/ceph/ceph-client.0.44700.log.gz 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43199.log.gz 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50669.log 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41537.log 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51312.log.gz 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47219.log 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41537.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50669.log.gz 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41537.log.gz 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53066.log 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43196.log 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47219.log.gz 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47015.log 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43196.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53066.log.gz 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43196.log.gz 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46831.log 2024-04-24T23:51:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47531.log 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47015.log.gz 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52564.log 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47531.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46831.log.gz 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47531.log.gz 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42526.log 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52251.log 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52564.log.gz 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47699.log 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.42526.log.gz 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52251.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52251.log.gz 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53696.log 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45938.log 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47699.log.gz 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49464.log 2024-04-24T23:51:18.689 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45938.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53696.log.gz 2024-04-24T23:51:18.690 INFO:teuthology.orchestra.run.smithi016.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45938.log.gz 2024-04-24T23:51:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40576.log 2024-04-24T23:51:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45644.log 2024-04-24T23:51:18.690 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49464.log.gz/var/log/ceph/ceph-client.admin.40576.log: 2024-04-24T23:51:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53528.log 2024-04-24T23:51:18.690 INFO:teuthology.orchestra.run.smithi016.stderr: 88.2%/var/log/ceph/ceph-client.admin.45644.log: -- replaced with /var/log/ceph/ceph-client.admin.40576.log.gz 2024-04-24T23:51:18.690 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45644.log.gz 2024-04-24T23:51:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50399.log 2024-04-24T23:51:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49544.log 2024-04-24T23:51:18.690 INFO:teuthology.orchestra.run.smithi016.stderr: 89.1%/var/log/ceph/ceph-client.admin.50399.log: -- replaced with /var/log/ceph/ceph-client.admin.53528.log.gz 2024-04-24T23:51:18.690 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50399.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52928.log 2024-04-24T23:51:18.690 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-24T23:51:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49544.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49544.log.gz 2024-04-24T23:51:18.690 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50046.log 2024-04-24T23:51:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49128.log 2024-04-24T23:51:18.691 INFO:teuthology.orchestra.run.smithi016.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52928.log.gz 2024-04-24T23:51:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42009.log 2024-04-24T23:51:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49128.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50046.log.gz 2024-04-24T23:51:18.691 INFO:teuthology.orchestra.run.smithi016.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49128.log.gz 2024-04-24T23:51:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43929.log 2024-04-24T23:51:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42093.log 2024-04-24T23:51:18.691 INFO:teuthology.orchestra.run.smithi016.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42009.log.gz 2024-04-24T23:51:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43929.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43929.log.gz 2024-04-24T23:51:18.691 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42093.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42093.log.gz 2024-04-24T23:51:18.691 INFO:teuthology.orchestra.run.smithi016.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-04-24T23:51:18.778 INFO:teuthology.orchestra.run.smithi016.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-04-24T23:51:19.136 INFO:teuthology.orchestra.run.smithi016.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-04-24T23:51:19.185 INFO:teuthology.orchestra.run.smithi156.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-04-24T23:51:19.191 INFO:teuthology.orchestra.run.smithi156.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-04-24T23:51:19.209 INFO:teuthology.orchestra.run.smithi156.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-04-24T23:51:19.213 INFO:teuthology.orchestra.run.smithi016.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-04-24T23:51:19.214 INFO:teuthology.orchestra.run.smithi156.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-04-24T23:51:19.216 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-04-24T23:51:19.216 INFO:teuthology.orchestra.run.smithi156.stderr:real 0m0.719s 2024-04-24T23:51:19.216 INFO:teuthology.orchestra.run.smithi156.stderr:user 0m2.976s 2024-04-24T23:51:19.216 INFO:teuthology.orchestra.run.smithi156.stderr:sys 0m0.126s 2024-04-24T23:51:19.277 INFO:teuthology.orchestra.run.smithi016.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-04-24T23:51:19.284 INFO:teuthology.orchestra.run.smithi016.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-04-24T23:51:19.286 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-04-24T23:51:19.286 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m0.811s 2024-04-24T23:51:19.286 INFO:teuthology.orchestra.run.smithi016.stderr:user 0m3.191s 2024-04-24T23:51:19.286 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m0.270s 2024-04-24T23:51:19.287 INFO:tasks.ceph:Archiving logs... 2024-04-24T23:51:19.287 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219/remote/smithi016/log 2024-04-24T23:51:19.288 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T23:51:20.017 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219/remote/smithi156/log 2024-04-24T23:51:20.018 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T23:51:20.468 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-24T23:51:20.478 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-24T23:51:20.478 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T23:51:20.480 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T23:51:20.563 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T23:51:20.572 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T23:51:20.572 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:51:20.575 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T23:51:20.589 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-04-24T23:51:20.590 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-04-24T23:51:20.594 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:51:20.594 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-04-24T23:51:20.594 INFO:teuthology.orchestra.run.smithi016.stdout:^- hv01.front.sepia.ceph.com 2 6 377 55 +928us[ +938us] +/- 571ms 2024-04-24T23:51:20.594 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 +103us[ +114us] +/- 62ms 2024-04-24T23:51:20.594 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 -109us[ -109us] +/- 65ms 2024-04-24T23:51:20.594 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:51:20.595 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T23:51:20.595 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-04-24T23:51:20.596 INFO:teuthology.orchestra.run.smithi156.stdout:^- hv01.front.sepia.ceph.com 2 6 377 53 +931us[ +931us] +/- 571ms 2024-04-24T23:51:20.596 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 +100us[ +99us] +/- 62ms 2024-04-24T23:51:20.596 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 -104us[ -105us] +/- 65ms 2024-04-24T23:51:20.596 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T23:51:20.596 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T23:51:20.606 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T23:51:20.607 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T23:51:20.615 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T23:51:20.712 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T23:51:20.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:51:20.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:51:20.806 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T23:51:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:51:20.834 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T23:51:20.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T23:51:20.853 INFO:teuthology.task.internal:Duration was 532.064616 seconds 2024-04-24T23:51:20.853 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T23:51:20.863 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T23:51:20.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T23:51:20.865 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T23:51:20.903 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:51:20.910 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T23:51:21.273 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T23:51:21.273 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-04-24T23:51:21.273 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T23:51:21.296 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2024-04-24T23:51:21.297 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T23:51:21.341 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T23:51:21.341 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T23:51:21.344 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T23:51:21.406 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T23:51:21.406 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T23:51:21.412 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T23:51:21.543 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T23:51:21.555 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T23:51:21.555 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T23:51:21.580 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T23:51:21.605 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T23:51:21.616 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:51:21.624 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:51:21.644 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-04-24T23:51:21.667 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2024-04-24T23:51:21.679 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:51:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:51:21.712 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T23:51:21.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T23:51:21.735 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T23:51:21.746 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T23:51:21.747 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219/remote/smithi016 2024-04-24T23:51:21.747 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T23:51:21.795 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-24_21:24:02-fs-squid-distro-default-smithi/7672219/remote/smithi156 2024-04-24T23:51:21.795 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T23:51:21.831 INFO:teuthology.task.internal:Removing archive directory... 2024-04-24T23:51:21.832 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T23:51:21.834 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T23:51:21.879 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T23:51:21.892 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T23:51:21.893 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T23:51:21.903 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T23:51:21.903 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T23:51:21.905 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T23:51:21.918 INFO:teuthology.orchestra.run.smithi016.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 23:51 /home/ubuntu/cephtest 2024-04-24T23:51:21.933 INFO:teuthology.orchestra.run.smithi156.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 23:51 /home/ubuntu/cephtest 2024-04-24T23:51:21.934 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T23:51:21.946 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T23:51:21.971 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/recovery-fs} duration: 532.0646159648895 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-04-24T23:51:21.972 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T23:51:22.058 INFO:teuthology.run:pass